Add TryExtend trait

[bschoenmaeckers]

This PR adds a TryExtend trait that is similar to the std::iter::Extend trait except it is failable. Currently implemented for PyList & PyDict, including some optimised specializations on nightly.

[bschoenmaeckers]

I could not find a obvious file to add this trait to so I created a new one. Suggestions are welcome.

[mejrs]

I'm unsure whether I'm in favor of this, but I don't want specialization. We used to rely on it but removed it a couple of years ago. The problem with specialization (even the min version) is that it has soundness issues when it comes to lifetimes and it also has no realistic path towards stabilization.

Also, you can add a compatibility shim for PyList_Extend in the compat module.

[bschoenmaeckers]

Alternatively I could add 2 methods to PyList & PyDict; extend_from_pyobject and extend_from_iter. This does not require specialization and is optimized for both cases.

[codspeed-hq]

CodSpeed Performance Report

Merging #4664 will not alter performance

_{Comparing bschoenmaeckers:try-extend (bc7cff0) with main (5464f16)}

Summary

✅ 83 untouched benchmarks

[davidhewitt]

Sorry to have been a little slow to get around to joining the discussion on this PR, so many parallel threads at the moment 😩

Thanks very much for working on our collections! I am 100% in agreement that there are things we could do to make PyO3's wrappers of Python collections interact better with Rust iterators. I have been wondering about whether we could implement both FromIterator and regular Extend for a while; I was wondering if we could support iterators of Bound<'py, T> for those traits. I'd love your take on those possibilities and how / why TryExtend is necessary instead.

I am a little cautious about TryExtend being a trait which seems to be much broader than just scoped to Python domain. I wonder, why is it that this isn't in something like itertools or even std? Should we be trying to upstream it, and then having itertools as an optional feature, maybe?

[bschoenmaeckers]

I have been wondering about whether we could implement both FromIterator and regular Extend for a while;

I was hesitant to implement non-failable traits because this may cause unexpected panics when python throws an error back when creating a python collection. That said I do not know all cases when python fails adding an item, so maybe this is not that big of a problem.

A Extend & FromIterator implementation for PyList could look like this,

impl<'py, T> Extend<Bound<'py, T>> for Bound<'_, PyList>
{
    fn extend<I>(&mut self, iter: I)
    where
        I: IntoIterator<Item = Bound<'py, T>>,
    {
        iter.into_iter().for_each(|item| {
            self.append(item).expect("Failed to append item to list");
        });
    }
}

impl<'py, T> FromIterator<Bound<'py, T>> for Bound<'py, PyList> {
    fn from_iter<I: IntoIterator<Item=Bound<'py, T>>>(iter: I) -> Self {
        let mut iter = iter.into_iter().peekable();
        let py = iter.peek().map(Bound::py)
            .expect("Cannot create a list from an empty iterator");
        let mut list = PyList::empty(py);
        list.extend(iter);
        list
    }
}

Furthermore the FromIterator implementation requires that the iterator is not empty so we can get a GIL token that lives long enough. This makes me question if we should even try to implement this trait.

I am a little cautious about TryExtend being a trait which seems to be much broader than just scoped to Python domain. I wonder, why is it that this isn't in something like itertools or even std? Should we be trying to upstream it, and then having itertools as an optional feature, maybe?

I like this idea, itertools has an ongoing discussion on fallible iterators so TryExtend maybe a good addition.