[pull] master from image-rs:master #7

Fix a bunch of warnings

Consolidate Avg remainder loop to main loop

Use iterators in Up and Avg filtering

Drastically improve encoding performance

Co-authored-by: okaneco <[email protected]>

Optimize adaptive filtering

Add an initial roundtrip fuzz target

Optimize Paeth decoding

Also roundtrip-fuzz with various bit depths

Use observations from the optimized Paeth encoding filter to do less work.

Simplify Paeth decode filter, fix remaining lib clippy lints

Use bitwise manipulation to find the average of two integers without casting to a wider integer type

The idea for SWAR is to treat regular integer types like `u32|u64` as a SIMD register for arbitrary smaller-width bit lanes. We can calculate 3 bytes per pixel and 4bpp pixels within a `u32`, 6bpp and 8bpp pixels within a `u64`. It's not an n-times speedup based on the number of lanes because SWAR may require some overhead of maintaining the lanes depending on the operation (shifts, for instance). This is evidenced by masking operations during calculation to correct or prevent lanes from overlapping. For the wrapping addition operation here, 7-bit arithmetic is calculated to prevent a carry digit affecting neighboring lanes. After the addition, a correcting XOR addition is done to repair the most significant bit and arrive at the correct answer for 8-bit wrapping unsigned arithmetic.

Add asserts to remove bounds checks

Move the check for `bpp > len` to `next_raw_interlaced_row` in `decoder`

…r_avg functions

Co-authored-by: Jonathan Behrens <[email protected]>

Refactor Avg unfilter to remove macro Refactor Paeth so all arms use chunks method Reorder match arms in unfilter for consistency

Optimize Sub, Avg, and Paeth decoding with SIMD Within A Register

This reuses some buffers that have previously been deallocated in each loop iteration. Additionally clippy has been showing me a bunch of warnings that I haved fixed as well.

Reuse some buffers and apply clippy fixes

Improve corpus-bench benchmark

Faster decoding of fdeflate encoded PNGs

Add support for palette in fuzz roundtrip

Add `ignore_adler32` flag to `ZlibStream` Disable computing and checking the Adler32 checksum by default Reset ZlibStream where possible instead of instantiating new structs

Rename `DecodeConfig` to `DecodeOptions` Remove `DecodeOptions` from `Decoder` Add `ignore_adler32` flag to `DecodeOptions`

Add ignore_checksums setter to Decoder Avoid updating and calculating CRC if flag is set to ignore

Refactor Reader implementation

Add option to ignore Adler32 checksum in ZlibStream, set ignore as default

Faster decompression of non-fdeflate encoded PNGs

Release 0.17.8-rc and call for testing

Release 0.17.8

Add rust-version to Cargo.toml

Improve unfilter() performance

Without this fix the icc profile would usually be empty

Release 0.17.9

More refactoring

Add alpha transformation

Signed-off-by: David Korczynski <[email protected]>

Add CIFuzz GitHub action

Remove copy-pasted line of documentation in set_ignore_crc

Put Crc update behind ignore_crc flag in StreamingDecoder::next_state

Release 0.17.10

Use Cargo.lock.msrv in CI

Remove copy_from_slice in palette match arms of next_interlaced_row_impl Reuse the previous row in unpack_bits calculation instead of expanding the palette within the buffer Use chunks_exact and iterate from start to end of the buffer instead of back to front in-place

Add assert for `bit_depth` in `unpack_bits` Special case for `BitDepth::Eight`

Ensure that the input buffer can produce enough bit shifts per input entry to match or exceed the length of the output buffer.

Refactor `utils::unpack_bits` for palette expanded images

Fixes image-rs/image#1825

…p-chunks Ignore subsequent iCCP chunks to match `libpng` behavior.

Scaffolding for direct benchmarking of `crate::filter::unfilter`.

Fixed `cargo afl build` compile error

Write document of `StreamWriter::finish`

…p=6 (#414)

This eliminate Reader::prev and adds special handling of unfiltering for the first row.

This helps with unifying how image data from IDAT and fdAT is handled. This unification is a prerequisite for the follow-up commit where we will avoid temporarily copying compressed image data into `current_chunk.raw_bytes`.

…bytes`. Before these changes we would store all image data in `ChunkState::raw_bytes` before attempting to decompress them via `ZlibStream`. This is undesirable for performance, because 1) it copies data unnecessarily into and out of `raw_bytes` and 2) by the time we have stored all the image data, the start of it may be already gone from the L1 cache. After these changes, the image data is passed directly into `ZlibStream`. This commit is part of a bigger set of changes that avoids copying image pixels across different buffers. This commit alone gives notable although relatively small performance improvements, but the real motivation here is to make progress toward merging the bigger set of changes and realizing their performance gains. For more information see the "Revisiting copy avoidance" part of the discussion at #416 (comment) Performance impact of this commit (2 measurement attempts): * `decode/8x8-noncompressed.png` - runtime improved by 9.18% - 9.99% * `128x128-noncompressed.png` - runtime improved by 7.89% - 8.51% * `decode/Transparency.png` - runtime improved by 1.92% - 2.69% No impact on the other test inputs: * `decode/kodim02.png` - change within noise threshold (same result when rerun) * `decode/kodim07.png` - no change detected (p = 0.06 or 0.17 > 0.05) * `decode/kodim17.png` - runtime improved by 1.75% (change within noise threshold when rerun) * `decode/kodim23.png` - runtime regressed by 2.15% (no change detected when rerun - p = 0.86 > 0.05) * `Lohengrin_-_Illustrated_Sporting_and_Dramatic_News.png` - change within noise threshold (no change detected when rerun - p = 0.14 > 0.05)

The move will help with follow-up work, where we want to use generated PNGs in unit tests of `src/decoder/stream.rs`.

This has accidentally regressed in a recent commit and was caught during PR review.

Pass image data directly to `ZlibStream`, bypassing `ChunkState::raw_bytes`

Skipping non-critical chunks if the CRC check fails is the default it libpng. Therefore adding this option and enabling it by default. image-rs/image#2057

Co-authored-by: hoijui <[email protected]>

Import SimdOrd, SimdPartialEq from std::simd::cmp Import SimdInt, SimdUint from std::simd::num The most recent portable-simd subtree sync moved some traits out of the top level modules.

This way removes a separate `State::Signature` variant and handles the PNG signature via two `State::U32` states. This is desirable because: * This helps to reuse the performance gains for `State::U32` (i.e. avoid accumulating the signature byte-by-byte if the input buffer already has all its bytes) * Avoiding separate code also has additional benefits: - Less pressure on the instructions cache - Ability to reuse branch prediction for `U32`-related code The modified code already has reasonable test coverage via http://www.schaik.com/pngsuite/#corrupted

* Roll the `fdeflate` dependency to version 0.3.3 * Cap buffer sizes via `ZlibStream::set_max_total_output`. Before this commit, `ZlibStream::new` would always allocate and zero out 64kB of data via `out_buffer: vec![0; 2 * CHUNCK_BUFFER_SIZE]`. After this commit, `out_buffer` is capped by the actual size of the decompressed image data. `StreamingDecoder::parse_ihdr` estimates the image size and notifies `ZlibStream` via the new `set_max_total_output` method. Impact on the runtime of the `decode/generated-png:noncompressed-8x8.png` benchmark (3 measurements): * [-24.028% -23.693% -23.460%] (p = 0.00 < 0.05) * [-23.830% -23.480% -23.200%] (p = 0.00 < 0.05) * [-21.259% -20.893% -20.618%] (p = 0.00 < 0.05)

Avoiding 32kB decompression lag =============================== Before this commit, decompressed data would be accumulated in `ZlibStream::out_buffer` and returned via `image_data` with 32kB lag corresponding to `CHUNCK_BUFFER_SIZE`: ``` fn transfer_finished_data(&mut self, image_data: &mut Vec<u8>) -> usize { let safe = self.out_pos.saturating_sub(CHUNCK_BUFFER_SIZE); image_data.extend(self.out_buffer.drain(..safe)); ... ``` 32kB is a typical size of L1 cache, so the lag would mean that the data passed to `image_data.extend(...)` would be already cold and evicted from the L1 cache. This commit avoids the lag by always returning into `image_data` all the data from `out_buffer` (i.e. data up to `out_pos`): ``` fn transfer_finished_data(&mut self, image_data: &mut Vec<u8>) -> usize { let transferred = &self.out_buffer[self.read_pos..self.out_pos]; image_data.extend_from_slice(transferred); self.read_pos = self.out_pos; ... ``` Compacting less often ===================== The changes above mean that `Vec::drain` no longer compacts `out_buffer`. Therefore this commit also refactors how this compaction works. Before this commit, not-yet-returned data would be shifted to the beginning of `out_buffer` every time `transfer_finished_data` is called. This could potentially mean that for 1 returned byte, N bytes have to be copied during compaction. After this commit, compaction is only done when the compaction cost if offset by many read bytes - for 3 returned bytes 1 byte has to be copied during compaction. Performance impact ================== The commit has a positive impact on performance, except for: * `decode/Transparency.png` - regression between 15% and 20% is reported in 3-out-of-3 measurements. * `decode/kodim17.png` - a regression of 2.1% has been reported in 1-out-of-3 measurements (an improvement of 0.6% - 1.13% has been reported in the other 2-out-of-3 measurements). * `generated-noncompressed-64k-idat/128x128.png` - a regression of 25% has been reported in 1-out-of-3 measurements (an improvement of 21% - 29% has been reported in the other 2-out-of-3 measurements). The results below have been gathered by running the `decoder` benchmark. First a baseline was saved before this commit, and then a comparison was done after the commit. This (the baseline + the comparison) was repeated a total of 3 times. All results below are for the relative impact on the runtime. All results are with p = 0.00 < 0.05. * decode/kodim23.png: * [-2.9560% -2.7112% -2.4009%] * [-3.4876% -3.3406% -3.1928%] * [-3.0559% -2.9208% -2.7787%] * decode/kodim07.png: * [-1.2527% -1.0110% -0.6780%] * [-1.7851% -1.6558% -1.5164%] * [-1.6576% -1.5216% -1.3856%] * decode/kodim02.png: * [-0.5108% -0.2806% -0.0112%] * [-1.0885% -0.9493% -0.8118%] * [-0.5563% -0.4239% -0.2874%] * decode/kodim17.png: * [+1.8649% +2.1138% +2.4169%] (**regression**) * [-1.2891% -1.1322% -0.9736%] * [-0.7753% -0.6276% -0.4866%] * decode/Lohengrin_-_Illustrated_Sporting_and_Dramatic_News.png: * [-1.7165% -1.4968% -1.2650%] * [-1.7051% -1.4473% -1.2229%] * [-1.2544% -1.0457% -0.8375%] * decode/Transparency.png: * [+19.329% +19.789% +20.199%] (**regression**) * [+15.337% +15.798% +16.294%] (**regression**) * [+18.694% +19.106% +19.518%] (**regression**) * generated-noncompressed-4k-idat/8x8.png: * [-2.3295% -1.9940% -1.5912%] * [-6.1285% -5.8872% -5.6091%] * [-2.8814% -2.6787% -2.4820%] * generated-noncompressed-4k-idat/128x128.png: * [-59.793% -59.599% -59.417%] * [-63.930% -63.846% -63.756%] * [-62.377% -62.248% -62.104%] * generated-noncompressed-4k-idat/2048x2048.png: * [-67.678% -67.579% -67.480%] * [-65.616% -65.519% -65.429%] * [-65.824% -65.647% -65.413%] * generated-noncompressed-4k-idat/12288x12288.png: * [-60.932% -60.774% -60.528%] * [-62.088% -62.016% -61.940%] * [-61.663% -61.604% -61.546%] * generated-noncompressed-64k-idat/128x128.png: * [-22.237% -21.975% -21.701%] * [-29.656% -29.480% -29.311%] * [+24.812% +25.190% +25.571%] (**regression**) * generated-noncompressed-64k-idat/2048x2048.png: * [-21.826% -21.499% -21.087%] * [-54.279% -54.049% -53.715%] * [-11.174% -10.828% -10.482%] * generated-noncompressed-64k-idat/12288x12288.png: * [-40.421% -40.311% -40.180%] * [-39.496% -39.183% -38.871%] * [-41.443% -41.367% -41.295%] * generated-noncompressed-2g-idat/2048x2048.png: * [-40.136% -40.010% -39.865%] * [-58.507% -58.333% -58.060%] * [-35.822% -35.457% -35.038%] * generated-noncompressed-2g-idat/12288x12288.png: * [-37.196% -37.107% -37.014%] * [-36.125% -36.049% -35.970%] * [-35.636% -35.477% -35.350%] Co-authored-by: Lukasz Anforowicz <lukasza@marcin-mx>

…er the image (#450)

Co-authored-by: Jonathan Behrens <[email protected]>

This commit introduces a new `transform.rs` module and moves row transformation functions into the new module: * From `util.rs`: - `unpack_bits` (no longer needs to be `pub`) - `expand_trns_line` - `expand_trns_line16` - `expand_trns_and_strip_line16` * From `mod.rs`: - `expand_paletted` - `expand_gray_u8` This commit also renames `util.rs` into `adam7.rs`, because after the refactoring above this module contains only Adam7-related functionality: - `struct Adam7Iterator` - `fn expand_pass` which operates on already-transformed, but still-interlaced rows) This commit is intended to be just pure refactoring (i.e. no changes in behavior or performance are expected).

This ensures that all the public functions in the `transform.rs` module are infallible.

This commit tweaks `transform.rs` so that all the functions take the same parameters: `input: &[u8], output: &mut [u8], info: &Info`. This is achieved by: 1. Taking `info: &Info` instead of `trns: Option<&[u8]>, channels: usize` for `expand_trns_line`, `expand_trns_line16`, `expand_trns_and_strip_line16`. 2. Removing `trns: Option<Option<&[u8]>>` parameter from `expand_paletted` and `expand_gray_u8` by splitting these functions into two separate flavors: ones that emit an alpha channel and ones that don't.

Instead of deciding which function to use for every row, memoize and reuse the first decision. One desirable outcome of this commit is making the public API of the `transform.rs` module quite thin (just the `TransformFn` type alias and the `create_transform_fn` function) - this makes this functionality easier to test and benchmark. Another desirable outcome is a small runtime improvement in most benchmarks (compared to the baseline just before the commit that introduces `transform.rs`): decode/paletted-zune.png: [-8.7989% -7.4940% -6.1466%] (p = 0.00 < 0.05) decode/kodim02.png: [-4.4824% -4.0883% -3.6232%] (p = 0.00 < 0.05) decode/Transparency.png: [-4.5886% -3.5213% -2.2121%] (p = 0.00 < 0.05) decode/kodim17.png: [-2.4406% -2.0663% -1.7093%] (p = 0.00 < 0.05) decode/kodim07.png: [-3.4461% -2.8264% -2.2676%] (p = 0.00 < 0.05) decode/kodim23.png: [-1.7490% -1.3101% -0.7639%] (p = 0.00 < 0.05) decode/Lohengrin: [-2.9387% -2.3664% -1.7545%] (p = 0.00 < 0.05) generated-noncompressed-4k-idat/8x8.png: [-4.0353% -3.5931% -3.1529%] (p = 0.00 < 0.05) generated-noncompressed-4k-idat/128x128.png: [-5.2607% -4.6452% -4.0279%] (p = 0.00 < 0.05) generated-noncompressed-4k-idat/2048x2048.png: [-3.0347% -1.7376% -0.4028%] (p = 0.03 < 0.05) generated-noncompressed-64k-idat/128x128.png: [-2.3769% -1.7924% -1.2211%] (p = 0.00 < 0.05) generated-noncompressed-64k-idat/2048x2048.png: [-12.113% -9.8099% -7.2633%] (p = 0.00 < 0.05) generated-noncompressed-64k-idat/12288x12288.png: [-5.0077% -1.4750% +1.4708%] (p = 0.43 > 0.05) generated-noncompressed-2g-idat/12288x12288.png: [-9.1860% -8.2857% -7.3934%] (p = 0.00 < 0.05) Some regressions were observed in 2 benchmarks: generated-noncompressed-4k-idat/12288x12288.png: [+2.5010% +3.1616% +3.8445%] (p = 0.00 < 0.05) [+3.6046% +4.6592% +5.8580%] (p = 0.00 < 0.05) [+4.6484% +5.4718% +6.4193%] (p = 0.00 < 0.05) generated-noncompressed-2g-idat/2048x2048.png: [-0.6455% +1.9676% +3.9191%] (p = 0.13 > 0.05) [+6.7491% +8.4227% +10.791%] (p = 0.00 < 0.05) [+5.9926% +7.2249% +8.5428%] (p = 0.00 < 0.05)

This commit moves `expand_paletted_into_rgb8` and `expand_paletted_into_rgba8` (and their unit tests) into a separate `transform/palette.rs` module. This prepares room for encapsulating extra complexity in this module in follow-up commits, where we will start to precompute and memoize some data when creating a `TransformFn`. This commit just moves the code around - it should have no impact on correctness or performance.

The `PLTE` chunk's size should be a multiple of 3 (since it contains RGB entries - 3 bytes per entry). Additionally, taking 10000 samples in the `bench_create_fn` benchmarks is a bit excessive after memoization.

This commit changes the `TransformFn` type alias from `fn(...)` into `Box<dyn Fn(...)>`. This allows the `TransformFn` to have store some precomputer, memoized state that we plan to add in follow-up commits. In theory this commit may have negative performance impact, but in the grand scheme of things it disappears into the measurement noise. In particular, when there is no state, then `Box` shouldn't allocate.

Before this commit `expand_paletted_into_rgba8` would: * Perform 2 lookups - `palette.get(i)` and `trns.get(i)` * Check via `unwrap_or` if `i` was within the bounds of `palette`/`trns` This commit introduces `create_rgba_palette` which combines `palette` and `trns` into a fixed-size `[[u8;4]; 256]` look-up table (called `rgba_palette` in the code). After this commit `expand_paletted_into_rgba8` only needs to perform a single look-up and doesn't need to check the bounds. This helps to improve the expansion time by 60+%: - expand_paletted(exec)/trns=yes/src_bits=4/src_size=5461: [-60.208% -60.057% -59.899%] (p = 0.00 < 0.05) - expand_paletted(exec)/trns=yes/src_bits=8/src_size=5461: [-77.520% -77.407% -77.301%] (p = 0.00 < 0.05) `expand_paletted_into_rgb8` performs only a single lookup before and after this commit, but avoiding bounds checks still helps to improve the expansion time by ~12%: - expand_paletted(exec)/trns=no/src_bits=4/src_size=5461: [-12.357% -12.005% -11.664%] (p = 0.00 < 0.05) - expand_paletted(exec)/trns=no/src_bits=8/src_size=5461: [-13.135% -12.584% -12.092%] (p = 0.00 < 0.05) Understandably, this commit regresses the time of `create_transform_fn`. Future commits will reduce this regression 2-4 times: - expand_paletted(ctor)/plte=256/trns=256: [+3757.2% +3763.8% +3770.5%] (p = 0.00 < 0.05) - expand_paletted(ctor)/plte=224/trns=32: [+3807.3% +3816.2% +3824.6%] (p = 0.00 < 0.05) - expand_paletted(ctor)/plte=16/trns=1: [+1672.0% +1675.0% +1678.1%] (p = 0.00 < 0.05)

Before this commit `expand_into_rgb8` would copy 3 bytes at a time into the output. After this commit it copies 4 bytes at a time (possibly cloberring pixels that will be populated during the next iteration - this is ok). This improved the performance as follows: expand_paletted(exec)/trns=no/src_bits=8/src_size=5461 time: [-23.852% -23.593% -23.319%] (p = 0.00 < 0.05)

This improves the performance as follows: - expand_paletted(ctor)/plte=256/trns=256 [-40.581% -40.396% -40.211%] (p = 0.00 < 0.05) - expand_paletted(ctor)/plte=224/trns=32 [-24.070% -23.840% -23.592%] (p = 0.00 < 0.05) Small palettes are mostly unaffected: - expand_paletted(ctor)/plte=16/trns=1 [-0.2525% +0.0338% +0.3239%] (p = 0.81 > 0.05)

Remove remaining uses of miniz_oxide for decoding

Remove usages of `extern crate`

Fixes #357.

Fixes #273.

Add SIMD-accelerated unfiltering for `BytesPerPixel::{Four, Eight}` Add `paeth_predictor_u8`, `paeth_step_u8`, and `unfilter_paeth_u8` for calculating the Paeth predictor entirely in `Simd<u8, N>` without converting to `Simd<i16, N>` Parameterize `PaethState` by `T`, the `SimdElement` type Calculating the filter entirely in `u8` avoids penalties which result from unpacking to and packing from `Simd<i16, N>`. `pa`, `pb`, and `pc` can all be calculated using absolute differences and reusing the logic of `filter::filter_paeth`.

Use SIMD to speed up Paeth unfiltering for bpp=4, bpp=8

Update miniz_oxide to 0.8.0.

This helps with the following things: * It means that after making `InterlaceInfo` public in a follow-up commit, `pass`, `line`, and `width` fields can remain private. * It means that a follow-up commit that refactors `adam7::expand_pass` to take a single `info` parameter doesn't allow passing an invalid `InterlaceInfo::Null` value. * It gives `Adam7Iterator` a nicer, named `Iterator::Item`.

Instead of passing `line_no` and `pass` as separate parameters, we can pass a single `&Adam7Info` parameter. This minor refactoring helps with: * Exposing a nice public API in a follow-up commit * Depending on `Adam7Info.width` in a follow-up commit

Instead of passing `line_no` and `pass` as separate parameters, we can pass a single `&Adam7Info` parameter. This minor refactoring helps with: * Cleaning up `fn expand_pass` by moving some of its complexity closer to where its needed (within `fn expand_adam7_bits`) * Preparing for changing the semantics of the `width` parameter of `fn expand_adam7_bits` in a follow-up commit, where we will stop depending on `width` for calculating the length of the returned iterator (depending on `Adam7Info.width` instead). See the follow-up commit for more details.

This commit refactors implementation of `expand_adam7_bits` so that it can accept `row_stride_in_bytes` that is different from the expanded width of the frame or image. This is helpful for making `expand_pass` work in scenarios where interlaced row needs to expanded into a bigger image (e.g. if the currently decoded frame is an animation frame that only takes a subregion of the whole image - in this case the stride between expanded rows is bigger than the size of expanded rows). And the above is helpful for exposing `expand_pass` through a public API that will hopefully pass the test of time...

This also exposes supporting items through the public API: `Adam7Info`, `InterlaceInfo`, `InterlacedRow`.

New public API `pub fn expand_interlaced_row`

This commit decouples `fn next_pass` and `InterlaceIter` from the rest of `mod.rs` - this coupling was mostly limited to 1) using `self.subframe.rowlen` for `InterfaceInfo::Null` and 2) resetting `self.prev_start` in-between Adam7 passes. This decoupling allows to encapsulate the related functionality inside a new `interlace_info.rs` module. This helps in follow-up commits, where we want to refactor when exactly `Reader` advances that iterator.

This commit supports detecting unexpected EOF using the public API of of `DecodingError`. Before this commit `UnexpectedEof`, `UnexpectedEndOfChunk`, and `NoMoreImageData` errors were represented as a crate-internal `FormatErrorInner` type. After this commit, these errors have a representation that can be detected using the public API: `DecodingError::IoError(std::io::ErrorKind::UnexpectedEof.into())`.

This commit supports resuming decoding after `UnexpectedEof`: * Before this commit `fn next_interlaced_row` would unconditionally call `InterlaceInfoIter.next` - advancing to the next row. After this commit this will only happen after a row has been successfully decoded (inside `next_interlaced_row_impl` because advancing needs to be synchronized across the public `next_frame` and `next_interlaced_row` APIs).. * Before this commit `fn next_frame` would always start decoding into the very beginning of `buf`, even if these initial rows have already been successfully decoded earlier. After this commit an offset is calculated based on `InterlaceInfo.line_number`. * Before this commit `fn next_frame` would always reset `data_stream`, `current_start`, and `prev_start`. After this commit, this got moved and only happens inside `read_until_image_data`.

Before this commit, the `buf_independent.rs` fuzzer would be hardcoded to fuzz a pair of decoder implementations (`smal` and `reference`). After this commit, the fuzzer operates on an arbitrarily-long vector of decoder implementations. The vector contains homogenous item types because the actual `Read` type is hidden behind the `Box<dyn Read + 'a>` indirection. The motivation for this commit is the desire to add coverage of a 3rd kind of decoder implementation in a follow-up commit. This commit doesn't change the operations being tested (`Decoder.new_with_limits`, `Decoder.read_info`, multiple `Reader.next_frame`). This commit doesn't change what verification is being done (consistent `Ok`-vs-`Err` results, identical decoded buffers, identical `OutputInfo`). This commit opportunistically adds extra assertions that all `Info` structs contain the same data.

Support for resuming decoding after `UnexpectedEof`.

Fuzzing found an example of an animated PNG that would trigger an assert in `fn read_until_image_data`, indicating that some `image_data` is unexpectedly appended into a throw-away `buf`. This example is being added to `fuzz/corpus/buf_independent/regressions/...`. This example was originally reported at https://oss-fuzz.com/testcase-detail/6611224761008128 In the example, a sequence of `IDAT` chunks was interleaved with some other (ignored, unrecognized) chunks. During normal, non-interrupted decoding `fn next_frame` would advance to the invalid next sequence of consecutive chunks and decode them. OTOH when resuming decoding after `UnexpectedEof` `fn next_frame` would see that it is still not at the expected frame, and `fn read_until_image_data` would end up decoding some image data and failing `assert!(buf.is_empty())`. The problem described above is fixed by adding checks that an `IDAT` sequence may only start once per PNG file + that an `fdAT` sequence may only start if we've seen an `fcTL` chunk after the previous `fdAT` sequence has terminated.

A truncated PNG file can legitimately result in an `UnexpectedEof` error being reported. Before this commit, `fn retry_after_eofs` in the `buf_independent.rs` fuzzer would repeatedly try to resume such unresumable input. Fuzzing found an example that illustrates the problem described above. This example is being added to fuzz/corpus/buf_independent/regressions/.... This example was originally reported at https://oss-fuzz.com/testcase-detail/5790077345660928

…fuzzer.

#496 has incorrectly started to return `UnexpectedEof` in a situation when when there is no `Read`-level EOF. In particular, this may happen when an `IDAT` of `fdAT` chunk has been fully decoded, decompressed, and flushed, but still doesn't contain the expected number of image pixels. In such a situation the old `NoMoreImageData` error was appropriate. In a sense, this commit is a partial revert and/or refactoring of the earlier commit b5b0674 This commit fixes a timeout found locally by the `buf_independent` fuzzer. The repro case has been saved under `fuzz/corpus/buf_independent/regressions`.

Fuzzing found a broken PNG file where an `fcTL` chunk is too short, and where `read_be` or `read_exact` in `parse_fctl` returns `UnexpectedEof`. This file was saved under `fuzz/corpus/buf_independent/regressions`. Before this commit, the input above would mislead the client/fuzzer code into thinking that there is a recoverable error that may go away after more PNG bytes are available. (This was incorrect because `parse_chunk` and `parse_fctl` are only called after gathering *all* the bytes of a chunk into `ChunkState::raw_bytes`.) But when the client tried resuming, we got into an infinite loop because `StreamingDecoder` has already given up and set `state` to `None`. After this commit, the timeout is avoided by properly indicating a non-recoverable error in such a situation.

Fuzzing found a broken PNG file where the sequence of chunks looks more or less like this: `IHDR`, `acTL`, `IDAT`, `fcTL`, `fctL`, EOF. This commit preserves this repro file under `fuzz/corpus/buf_independent/regressions`. Before this commit such fuzzing input would lead to problems when resuming a call to `next_frame` after an interittent EOF. There were a few loosely related problems: 1. `fn read_until_image_data` would unnecessarily set `self.subframe` in the `Decoded::FrameControl` branch. This was unnecessary because `self.subframe` is also set after breaking out of the loop. 2. `fn read_until_image_data` is called to position the input stream at the beginning of the next image frame - it breaks out of its loop and returns after encountering the start of the next `IDAT` or `fdAT` chunk. This means that there is a discrepancy: - Whether we are in the middle of an `IDAT`/`fdAT` data stream can be determined by checking `self.subframe.consumed_and_flushed`. - But before this commit `fn next_frame` would decide whether to call `read_until_image_data` based on `self.next_frame`, rather than based on `self.subframe.consumed_and_flushed`. 3. After calling `fn finish_decoding` from `fn next_frame` the `self.subframe.consumed_and_flushed` wouldn't be reset. Problem2 above meant that after an intermittent `UnexpectedEof` in `read_until_image_data` retrying a call to `next_frame` would *not* retry `read_until_image_data` (because `self.next_frame` would be updated by the previous `read_until_image_data`, and now `fn next_frame` would incorrectly treat this as not needing to call `read_until_image_data`, even though we still haven't reached an `IDAT` nor `fdAT` chunk). This was fixed by changing how `fn next_frame` decides whether to call `read_until_image_data` (as a side-effect this allowed deleting `subframe_idx`). Fixing problem2 was insufficient, because problem1 meant that `consumed_and_flushed` was prematurely set (before actually encountering an `IDAT` or `fdAT` frame). And we also needed to fix problem3 to ensure that the next call to `next_frame` knows that it is time to advance to the next `IDAT` / `fdAT` chunks.

This is a follow-up to the feedback at #500 (comment)

… to documentation

Co-authored-by: Jonathan Behrens <[email protected]>

Improve documentation on filters

New release

…surprising performance gain

`unstable` feature doesn't affect encoding; + fix grammar

Replace handwritten SIMD implementation with autovectorization for +10% perf

AFAICT before this commit `cargo test` didn't cover the code path in `Reader.next_frame` that results in `PolledAfterEndOfImage` error. This commit adds unit tests that provide such coverage.

This commit adds a test with the main assertions commented out (because they would fail as-is). A fix comes in the follow-up commit.

Before this commit, `next_frame` would take care to read till the end of an `IDAT`/`fdAT` chunks sequence, by calling `self.decoder.finish_decoding` when `!self.subframe.consumed_and_flushed`. `next_row` wouldn't do this and therefore the next call to `next_frame` would be temporarily stuck on the previous frame (i.e. finishing decoding after the previous frame rather than advancing to the next frame). After this commit, the `finish_decoding` code is extracted to a separate helper function that is called by *both* `next_frame` and `next_interlaced_row` (once they detect that all rows of a frame have been already decoded). This commit fixes the `test_row_by_row_then_next_frame` unit test.

In an earlier commit 278b1d4 we stopped using `Reader.next_frame` for deciding whether to call `read_until_image_data` from `next_frame`. This commit goes one step further and removes the `next_frame` field entirely. There are no known cases where the previous code would result in incorrect behavior, but this commit still seems desirable: * It simplifies the code * It ensures that a single "end-of-IDAT/fdAT-sequence" event kind controls both 1) `consumed_and_flushed` state, and 2) counting remaining frames. (Before this commit `next_frame` would also be mutated after a separate "fcTL-encountered" event. And also after "end-of-IDAT/fdAT-sequence" but only from `next_frame` and not from `next_row`.)

When `StreamingDecoder` reports an error, it leaves `state` set to `None`. Before this commit, calling `next_frame` in this state would have led to an infinite loop: * `ReadDecoder::decode_next` would loop forever (`!self.at_eof` is true after an error) and would fail to make progress, because * When `StreamingDecoder::update` sees `state` saw set to `None` then before this commit it wouldn't enter the `next_state` loop and would immediately return no progress (`Ok((/* consumer bytes = */ 0, Decoded::Nothing))`). After this commit, `StreamingDecoder::update` checks if the `state` is `None` and treats this as an error.

Before this commit calling `Reader.finish` a 2nd time would return a random, accidental error (`UnexpectedEof`), because of how `StreamingDecoder`'s `state` is set after processing the `IEND` chunk. After this commit, `Reader.finish` will handle this condition explicitly, in a similar same way to how `next_frame` handles being called when we have already consumed all the frames.

After a recent commit, all public APIs of a `Reader` take care of not going beyond the `IEND` chunk and returning `PolledAfterEndOfImage` instead. This means that tracking `at_eof` at the level of `ReadDecoder` is obsolete and leads to unnecessary complexity. This commit refactors away this complexity.

`ReadDecoder::finish_decoding` now reuses `decode_next` instead of duplicating some of its code. Some duplication (e.g. handling of `Decoded::Nothing` remains - this will be taken care of in a subsequent commit).

This commit means that `decode_next` can be a private method of `ReadDecoder` (this is not enforced yet, before a subsequent commit moves `ReadDecoder` into a separate `mod`ule).

Before this commit `fn decode_next` would `loop` to skip `Decoded::Nothing` events. This is unnecessary, because all the callers of `decode_next` already account for `Decoded::Nothing` (explicitly or implicitly via a wildcard).

This commit helps to keep some aspects of `ReadDecoder` private (e.g. its fields, `decode_next` methods, etc.). This commit also means that `mod.rs` no longer directly depends on `Decoded` nor `StreamingDecoder`.

Simplify and encapsulate `ReadDecoder` implementation

Extract a separate `unfiltered_rows_buffer` module.

Add support for parsing the sBIT chunk in the decoder

Commits on Dec 30, 2022

Faster decoding of fdeflate encoded PNGs

fintelia committed Dec 30, 2022

Configuration menu

View commit details

Copy full SHA for f0ef6fd

Browse repository at this point

Copy the full SHA

f0ef6fd View commit details

Browse the repository at this point in the history

Commits on Jan 2, 2023

Improve comment

Shnatsel committed Jan 2, 2023

Configuration menu

View commit details

Copy full SHA for 04d55cc

Browse repository at this point

Copy the full SHA

04d55cc View commit details

Browse the repository at this point in the history

Commits on Jan 15, 2023

Add support for palette in fuzz

ogarokpeter committed Jan 15, 2023

Configuration menu

View commit details

Copy full SHA for d845db5

Browse repository at this point

Copy the full SHA

d845db5 View commit details

Browse the repository at this point in the history

Commits on Jan 21, 2023

Add comments summarizing this PR

okaneco committed Jan 21, 2023

Configuration menu

View commit details

Copy full SHA for 7d6fa6b

Browse repository at this point

Copy the full SHA

7d6fa6b View commit details

Browse the repository at this point in the history

Commits on Feb 4, 2023

Downgrade to clap 3.0 to retain MSRV

fintelia committed Feb 4, 2023

Configuration menu

View commit details

Copy full SHA for a46450c

Browse repository at this point

Copy the full SHA

a46450c View commit details

Browse the repository at this point in the history

Commits on Mar 10, 2023

Remove unused function

okaneco committed Mar 10, 2023

Configuration menu

View commit details

Copy full SHA for 92b55e8

Browse repository at this point

Copy the full SHA

92b55e8 View commit details

Browse the repository at this point in the history

Commits on Apr 3, 2023

Release 0.17.8

fintelia committed Apr 3, 2023

Configuration menu

View commit details

Copy full SHA for ad7e2d9

Browse repository at this point

Copy the full SHA

ad7e2d9 View commit details

Browse the repository at this point in the history

Commits on May 29, 2023

Release 0.17.9

fintelia committed May 29, 2023

Configuration menu

View commit details

Copy full SHA for 5795162

Browse repository at this point

Copy the full SHA

5795162 View commit details

Browse the repository at this point in the history

Commits on Oct 27, 2023

Write document of StreamWriter::finish

H1rono committed Oct 27, 2023

Configuration menu

View commit details

Copy full SHA for abbbd19

Browse repository at this point

Copy the full SHA

abbbd19 View commit details

Browse the repository at this point in the history

Commits on Feb 4, 2024

Return Info with static lifetime (#465 )

fintelia authored Feb 4, 2024

Configuration menu

View commit details

Copy full SHA for a6425ca

Browse repository at this point

Copy the full SHA

a6425ca View commit details

Browse the repository at this point in the history

Commits on Feb 11, 2024

Test minimal versions in CI (#467 )

fintelia authored Feb 11, 2024

Configuration menu

View commit details

Copy full SHA for ec8dbe9

Browse repository at this point

Copy the full SHA

ec8dbe9 View commit details

Browse the repository at this point in the history

Commits on Feb 16, 2024

Release 0.17.12 (#468 )

fintelia authored Feb 16, 2024

Configuration menu

View commit details

Copy full SHA for 59e04df

Browse repository at this point

Copy the full SHA

59e04df View commit details

Browse the repository at this point in the history

Commits on Feb 18, 2024

Fix Send bound on Reader (#471 )

fintelia authored Feb 18, 2024

Configuration menu

View commit details

Copy full SHA for c02bd7b

Browse repository at this point

Copy the full SHA

c02bd7b View commit details

Browse the repository at this point in the history

Commits on Feb 19, 2024

Release 0.17.13 (#472 )

fintelia authored Feb 19, 2024

Configuration menu

View commit details

Copy full SHA for b5b0d48

Browse repository at this point

Copy the full SHA

b5b0d48 View commit details

Browse the repository at this point in the history

Commits on Aug 24, 2024

Update miniz_oxide to 0.8.0.

khuey committed Aug 24, 2024

Configuration menu

View commit details

Copy full SHA for e38e7e1

Browse repository at this point

Copy the full SHA

e38e7e1 View commit details

Browse the repository at this point in the history

Commits on Nov 29, 2024

Add bKGD chunk parsing (#538 )

ocpkn authored Nov 29, 2024

Configuration menu

View commit details

Copy full SHA for 9020cd9

Browse repository at this point

Copy the full SHA

9020cd9 View commit details

Browse the repository at this point in the history

[pull] master from image-rs:master #7

Are you sure you want to change the base?

[pull] master from image-rs:master #7

Commits on Dec 24, 2022

Commits on Dec 25, 2022

Commits on Dec 30, 2022

Commits on Dec 31, 2022

Commits on Jan 1, 2023

Commits on Jan 2, 2023

Commits on Jan 5, 2023

Commits on Jan 7, 2023

Commits on Jan 14, 2023

Commits on Jan 15, 2023

Commits on Jan 18, 2023

Commits on Jan 19, 2023

Commits on Jan 20, 2023

Commits on Jan 21, 2023

Commits on Jan 22, 2023

Commits on Jan 30, 2023

Commits on Feb 3, 2023

Commits on Feb 4, 2023

Commits on Feb 7, 2023

Commits on Feb 9, 2023

Commits on Feb 15, 2023

Commits on Mar 2, 2023

Commits on Mar 3, 2023

Commits on Mar 5, 2023

Commits on Mar 6, 2023

Commits on Mar 10, 2023

Commits on Mar 12, 2023

Commits on Mar 13, 2023

Commits on Mar 17, 2023

Commits on Mar 18, 2023

Commits on Mar 21, 2023

Commits on Apr 3, 2023

Commits on Apr 11, 2023

Commits on Apr 13, 2023

Commits on Apr 19, 2023

Commits on Apr 20, 2023

Commits on Apr 24, 2023

Commits on May 29, 2023

Commits on Jun 10, 2023

Commits on Jun 11, 2023

Commits on Jul 11, 2023

Commits on Jul 18, 2023

Commits on Aug 11, 2023

Commits on Aug 12, 2023

Commits on Aug 18, 2023

Commits on Aug 20, 2023

Commits on Aug 27, 2023

Commits on Aug 31, 2023

Commits on Sep 1, 2023

Commits on Sep 5, 2023

Commits on Sep 21, 2023

Commits on Sep 22, 2023

Commits on Sep 23, 2023

Commits on Sep 24, 2023

Commits on Oct 27, 2023

Commits on Oct 30, 2023

Commits on Nov 2, 2023

Commits on Nov 4, 2023

Commits on Nov 9, 2023

Commits on Nov 14, 2023

Commits on Nov 16, 2023

Commits on Nov 26, 2023

Commits on Dec 2, 2023

Commits on Dec 5, 2023

Commits on Dec 11, 2023

Commits on Dec 16, 2023

Commits on Dec 17, 2023

Commits on Dec 25, 2023

Commits on Jan 3, 2024

Commits on Jan 6, 2024

Commits on Jan 7, 2024

Commits on Jan 12, 2024

Commits on Jan 13, 2024

Commits on Jan 20, 2024

Commits on Jan 27, 2024

Commits on Jan 29, 2024

Commits on Feb 2, 2024