Introduce pinned places to the language

[mkrasnitski]

What is this?

Recently, a series of blog posts were published by withoutboats¹² as well as Olivier Faure³ discussing some existing problems with the ergonomics of the std::pin::Pin API, which can cause difficulty for users encountering it for the first time, especially when delving into the weeds of async/await (for example, when manually implementing Future).

In particular, a few specific ergonomic challenges users currently face are:

1. Pinning something that lives on the stack requires calling a macro, pin::pin!.
2. Pinned pointers cannot be automatically reborrowed, rather this must be done manually via Pin::as_mut.
3. Low-level async code often gets populated with self: Pin<&mut Self> receivers for methods.
4. Safely implementing so-called "pin projection" either requires unsafe code and a fair amount of boilerplate⁴, or importing an external crate like pin-project/pin-project-lite which generates the plumbing for you using macros.
5. Pin projection also interferes with the Drop trait, given that the trait is not aware of Pin in any way (it was added to the language first). If you implement pin projection yourself, it's possible to write a Drop impl that violates the pinning guarantees, which is an annoying footgun.

These problems basically all stem from the fact that Pin is a library type without enhanced language support (other than its use in desugared async code). The proposed solution is to introduce the concept of pinned places to Rust, along with a new keyword that facilitates the use of this new feature.

Motivation

It has been a couple months since this idea was first presented to the wider public and the initial reactions⁵ seemed overwhelmingly positive, in a way that I would consider rare for a feature which introduces new syntax. Therefore, I feel that an RFC should be put together by someone substantially motivated to do so. I am not that person - for one, I am not sufficiently versed in compiler internals nor entrenched enough in the Rust project to trust myself to write (what would be my first) RFC for a feature such as this one, that is subtle and hard to get right.

Instead, I wish to open a space for discussion so that those more familiar with type system internals can voice design concerns for any potential RFC writer to consider. Think of this as a pre-pre-RFC.

Additionally, though I suspect it may already be too late to do so, reserving a keyword for this in the 2024 edition would allow the feature to land sooner, rather than being blocked on the 2027 edition.

Feature Overview

Please note that none of what follows is my own original idea. My aim is to summarize the ideas others have put forth and document them here to serve as reference going forward.

The first question to answer is the choice of keyword. While withoutboats chose pinned in their blogposts, Olivier's opinion (one that I and others share) is that pin is much nicer-looking, which the example snippets below hopefully illustrate. Given that a new keyword requires an edition, the fact that std occasionally uses pin as an identifier or function parameter should not really be an issue (unless there is some other issue at play preventing its use).

Next, by letting the pin keyword to appear in a few different contexts, we can solve each of the above-mentioned ergonomic problems as follows:

1. Pinning something locally using pin bindings

By allowing pin to be used when creating bindings, it becomes easy to create locally pinned data. What would normally look like:

fn make_stream() -> impl Stream<Item = u32> {
    // ...
}

let mut stream = pin::pin!(make_stream());

...would instead be accomplished by:

let pin mut stream = make_stream();

The pin keyword does not change the type of stream, just like mut, but unlike the pin! macro which gives a Pin<&mut T>. Instead, it marks the memory location (i.e. the place) that stream refers to as pinned, meaning you can't move out of it and can't take a regular mutable reference to it (because of mem::swap). If the type of stream implements Unpin, then those restrictions don't apply.

2. Language support for pinned references

Allowing the pin keyword to modify reference declarations allows the language to automatically perform reborrowing on behalf of the user. By adding new &pin T and &pin mut T types which desugar to Pin<&T> and Pin<&mut T> respectively, pinned references can be easily created:

let pin mut stream = make_stream();
let stream_ref: &pin mut _ = &pin mut stream; // Equivalently, `stream_ref` is of type `Pin<&mut _>`

Not only that, the compiler is now aware of pinned references and can implement automatic reborrowing for them the same way it does for &mut T, transforming this:

let mut stream: Pin<&mut _> = pin!(make_stream());
stream.as_mut().poll_next();
stream.as_mut().poll_next();

...into this:

let pin mut stream = make_stream();
(&pin mut stream).poll_next();
(&pin mut stream).poll_next();

However, having to make a new pinned reference each time is still a bit unergonomic. If the compiler incorporates pinned references into its method resolution and automatically inserts them where necessary, we could just write the following:

let pin mut stream = make_stream();
// The `poll_next` method takes a `self: Pin<&mut Self>` as its first argument, 
// so the compiler automatically creates a pinned mutable reference to `self`.
stream.poll_next();
stream.poll_next();

This nicely leads into the next point:

3. Pinned bindings and pinned references in function arguments

Supporting pinned references in method resolution can be built upon with a bit of syntactic sugar that gets rid of the need to write self: Pin<&mut Self> receivers. Namely, adding &pin self and &pin mut self syntax that desugars in the same way &pin syntax does for references. For example, we could rewrite the futures::Stream trait like this to replace the method receiver on poll_next:

pub trait Stream {
    type Item;

    fn poll_next(&pin mut self, cx: &mut Context<'_>) -> Poll<Option<Self::Item>>;
}

What would be even better is to support pinned function arguments in general, allowing both pin bindings and &pin references. As an example:

struct Foo { /* ... */ }
struct Bar { /* ... */ }

fn baz(pin mut foo: Foo, bar: &pin Bar) {
    // ...
}

Within the body of baz, we know that foo is an owned value that the compiler ensures is given to baz pinned, meaning it won't move after being moved into baz. Its type is Foo, as is normally the case with pin bindings. Whereas, bar is a pinned reference whose type desugars to Pin<&Bar>, and we know it points to already-pinned memory somewhere outside the scope of baz.

4. Safe Pin Projection

The pin keyword also enables opt-in pin projection on a per-field basis for a given type by acting as a field modifier when defining the struct:

struct Foo {
    pub pin bar: Bar,
    baz: Baz,
}

Then, if we have a pinned Foo, we can obtain a pinned reference to a Bar, as well as an unpinned reference to a Baz:

let pin mut foo = Foo { /* ... */ };
let bar_ref = &pin mut foo.bar;
let baz_ref = &mut foo.baz;

In order for this to be sound, Foo must not have an explicit implementation of Unpin - now, it could in fact actually be Unpin, but only via the autotrait mechanism, which would require that both Bar and Baz are also Unpin.

5. Safely destructing types with pinned fields

There is an additional requirement for enabling safe pin projection, which is that types with pinned fields must not move out of those fields inside a type's destructor. Therefore, if a type with pinned fields implements Drop, it must do so in the following way:

impl Drop for Foo {
    fn drop(&pin mut self) {
        // The type of `self` is `&pin mut Self` instead of `&mut Self`
    }
}

By giving the destructor a pinned reference, any field accesses will follow the rules of pin projection and prevent moving out of any pinned places. This may seem at first to be a breaking change to the trait - however, Drop is special in that directly calling Drop::drop is not allowed in Rust, and manual destruction can only be invoked with mem::drop. Therefore, conditionally changing the signature of the Drop trait won't cause breakage.

Additional ideas and limitations

1. Constructing pinned data

One current limitation with pinning in Rust, is that while it is possible to pin an already-owned value, and it is possible to return a pinned reference from a function (by returning Pin<&T>), it is not currently possible to return a locally pinned value from a function; rather, the pinning must be performed after the function call. As an example, consider a pin binding:

let pin mut stream = make_stream()

The stream is created inside make_stream and moved out when it is returned, and then pinned locally after that. To make this clearer, we can split the assignment in two:

let not_yet_pinned_stream = make_stream()
let pin mut stream = not_yet_pinned_stream;

Here, we take the result of make_stream, and then move it into place and pin it there. This highlights the fact that we can only pin the result of make_stream after calling it; what we get from the function is not yet pinned. But, there are cases where we would like to return owned, pinned data that lives on the stack.

In their blogpost, Olivier gave an example from the cxx crate: due to small string optimizations in C++, an owned CxxString cannot be moved in Rust. Therefore, constructing one on the stack requires pinning. At the same time, if an object requires a constructor function and cannot be initialized inline, then calling the constructor will involve a move of the return value. This is why a CxxString on the stack can only be created with the let_cxx_string! macro.

Solving this completely requires something akin to either "placement new", or move constructors. Still, one vital piece of the puzzle would be a way to signal that a function returns something pinned. Olivier's suggestion was to annotate the return type with pin like so:

impl CxxString {
    pub fn new(bytes: &[u8]) -> pin Self {
        // ...
    }
}

However, given that pin just by itself modifies a place (whereas &pin desugars to a type), writing pin Self seems inconsistent with that pattern. Instead, I propose writing it as pin fn, which acts like a modifier in the same way that pinned fields do for the purposes of projection:

impl CxxString {
    pub pin fn new(bytes: &[u8]) -> Self {
        // ...
    }
}

This makes it clear that the return type of the function is still just Self, and that it returns an owned, pinned CxxString.

2. Pinned smart pointers

This is not so much a limitation, but it is worth noting that &pin T and &pin mut T only desugar to Pin<&T> and Pin<&mut T>, which does not account for the occasional use of Pin<Box<T>>, Pin<Arc<T>>, and Pin<Rc<T>>, which can be created with the {Box,Arc,Rc}::pin methods, respectively. It is obviously still possible to transform these into pinned references using as_ref()/as_mut(), but perhaps it's worth considering if language support for this should be added or not. One option is a DerefPinned trait which would facilitate coercion:

trait DerefPinned: Deref {
    fn deref_pinned(&mut self) -> &pin mut Self::Target;
}

Another option is to make use of &pin syntax in some way for coercion.

Conclusion

This post grew to be quite long, but I wanted to give a good summary of the feature as I understand it. Overall I feel it would be a great boon to ergonomics when dealing with pinning, and hopefully someone feels inspired enough to write up an RFC for it. Special thanks to @Jules-Bertholet and @withoutboats for helping plant the seed; I wrote this of my own accord but their ideas motivated me to do so.

Footnotes

1. https://without.boats/blog/pin/ ↩

2. https://without.boats/blog/pinned-places/ ↩

3. https://poignardazur.github.io/2024/08/16/pinned-places/ ↩

4. https://doc.rust-lang.org/std/pin/index.html#projections-and-structural-pinning ↩

5. https://www.reddit.com/r/rust/comments/1eagjl7/pinned_places/ ↩

[PatchMixolydic]

cc rust-lang/rust#130494