[Bug]: Semantic(None, "invalid type: bytes, expected bytes")

[RCasatta]

Is there an existing issue for this?

• I have searched the existing issues

Code of Conduct

• I agree to follow this project's Code of Conduct

Current Behaviour

I am getting the error Semantic(None, "invalid type: bytes, expected bytes")

Expected Behaviour

deserialization works

Environment Information

Linux 6.5.4-76060504-generic #202309191142169599894322.04~070916d SMP PREEMPT_DYNAMIC Fri S x86_64 x86_64 x86_64 GNU/Linux

Steps To Reproduce

The error is launched from here

ciborium/ciborium/src/de/mod.rs

Line 379 in 2ac91ce

header => Err(header.expected("bytes")),

because the header is bytes but the len of scratch is not enough

I can reproduce the error in a downstream crate with a type implementing custom serde Serialize/Deserialize: ElementsProject/rust-elements@45bd2bf
This type is longer than 4k bytes

When I try to reproduce without the downstream crate using something like Wrap(vec) where vec is bigger than 4k bytes, things work because in this line:

ciborium/ciborium/src/de/mod.rs

Line 369 in 2ac91ce

Header::Bytes(Some(len)) if len <= self.scratch.len() => {

the scratch space has grown to accommodate the len of the bytes, (it is 8k even though it is initialized to 4k) while in the case of the error it's probably not growing for some reason I could not grasp yet.

[RCasatta]

Following code could reproduce:

#[cfg(test)]
mod tests {
    use std::fmt;

    use serde::{de, Deserialize, Serialize};

    #[derive(PartialEq, Eq, Debug)]
    struct Wrap(Vec<u8>);

    impl Wrap {
        fn from_slice(slice: &[u8]) -> Result<Wrap, String> {
            Ok(Wrap(slice.to_vec()))
        }
        fn serialize(&self) -> Vec<u8> {
            self.0.clone()
        }
    }

    impl Serialize for Wrap {
        fn serialize<S: ::serde::Serializer>(&self, s: S) -> Result<S::Ok, S::Error> {
            s.serialize_bytes(&self.serialize())
        }
    }

    impl<'de> Deserialize<'de> for Wrap {
        fn deserialize<D: ::serde::Deserializer<'de>>(d: D) -> Result<Wrap, D::Error> {
            d.deserialize_bytes(BytesVisitor::new("a bytestring", Wrap::from_slice))
        }
    }

    pub struct BytesVisitor<F> {
        expectation: &'static str,
        parse_fn: F,
    }

    impl<F, T, Err> BytesVisitor<F>
    where
        F: FnOnce(&[u8]) -> Result<T, Err>,
        Err: fmt::Display,
    {
        pub fn new(expectation: &'static str, parse_fn: F) -> Self {
            BytesVisitor {
                expectation,
                parse_fn,
            }
        }
    }

    impl<'de, F, T, Err> de::Visitor<'de> for BytesVisitor<F>
    where
        F: FnOnce(&[u8]) -> Result<T, Err>,
        Err: fmt::Display,
    {
        type Value = T;

        fn expecting(&self, formatter: &mut fmt::Formatter) -> fmt::Result {
            formatter.write_str(self.expectation)
        }

        fn visit_bytes<E: de::Error>(self, v: &[u8]) -> Result<Self::Value, E> {
            (self.parse_fn)(v).map_err(E::custom)
        }
    }

    #[test]
    fn ciborium() {
        let a = Wrap(vec![0u8; 5000]);
        let bytes = cib_vec(&a).unwrap();
        let b: Wrap = ciborium::from_reader(&bytes[..]).unwrap();
        assert_eq!(a, b);
    }

    fn cib_vec<S: Serialize>(s: &S) -> Result<Vec<u8>, ciborium::ser::Error<std::io::Error>> {
        let mut vec = vec![];
        ciborium::ser::into_writer(s, &mut vec)?;
        Ok(vec)
    }
}