A plan for the guide

src/SUMMARY.md

@@ -11,10 +11,35 @@
 
 - [Introduction](part-guide/intro.md)
 - [Concurrent programming](part-guide/concurrency.md)
-- [Async and Await](part-guide/async-await.md)
+- [Async and await](part-guide/async-await.md)
+- [Advanced async/await topics](part-guide/adv-async-await.md)
+- [IO and issues with blocking](part-guide/io.md)
+- [Concurrency primitives](part-guide/concurrency-primitives.md)
+- [Channels, locking, and synchronization](part-guide/sync.md)
+- [Tools for async programming](part-guide/tools.md)
+- [Destruction and clean-up](part-guide/dtors.md)
+- [Futures](part-guide/futures.md)
+- [Runtimes](part-guide/runtimes.md)
+- [Timers and signal handling](part-guide/times-signals.md)
+- [Async iterators (streams)](part-guide/streams.md)
 
 # Part 2: reference
 
+- [Implementing futures and streams]()
+- [Alternate runtimes]()
+- [Implementing your own runtime]()
+- [async in sync, sync in async]()
+- [Async IO: readiness vs completion, and io_uring]()
+- [Design patterns]()
+- [Cancellation]() (cancellation safety)
+- [Starvation]()
+- [Pinning]()
+- [Async and FFI]()
+- [Comparing async programming in Rust to other languages]()
+- [The implementation of async/await in rustc]()
+- structured concurrency?
+
+
 # Old chapters
 
 - [Getting Started](01_getting_started/01_chapter.md)

src/part-guide/adv-async-await.md

@@ -0,0 +1,65 @@
+# More async/await topics
+
+## Unit tests
+
+## Blocking and cancellation
+
+- Two important concepts to be aware of early, we'll revisit in more detail as we go along
+- Cancellation
+  - How to do it
+    - drop a future
+    - cancellation token
+    - abort functions
+  - Why it matters, cancellation safety (forward ref)
+- Blocking
+  - IO and computation can block
+  - why it's bad
+  - how to deal is a forward ref to io chapter
+
+## `Send + 'static` bounds on futures
+
+- Why they're there, multi-threaded runtimes
+- spawn local to avoid them
+- What makes an async fn `Send + 'static` and how to fix bugs with it
+
+## Async traits
+
+- syntax
+  - The `Send + 'static` issue and working around it
+    - trait_variant
+    - explicit future
+    - return type notation (https://blog.rust-lang.org/inside-rust/2024/09/26/rtn-call-for-testing.html)
+- overriding
+  - future vs async notation for methods
+- object safety
+- capture rules (https://blog.rust-lang.org/2024/09/05/impl-trait-capture-rules.html)
+- history and async-trait crate
+
+
+## Async blocks and closures
+
+- async block syntax
+  - what it means
+- using an async block in a function returning a future
+  - subtype of async method
+- closures
+  - coming soon (https://github.com/rust-lang/rust/pull/132706, https://blog.rust-lang.org/inside-rust/2024/08/09/async-closures-call-for-testing.html)
+  - async blocks in closures vs async closures
+- errors in async blocks
+  - https://rust-lang.github.io/async-book/07_workarounds/02_err_in_async_blocks.html
+
+## Recursion
+
+- Allowed (relatively new), but requires some explicit boxing
+  - forward reference to futures, pinning
+  - https://rust-lang.github.io/async-book/07_workarounds/04_recursion.html
+  - https://blog.rust-lang.org/2024/03/21/Rust-1.77.0.html#support-for-recursion-in-async-fn
+  - async-recursion macro (https://docs.rs/async-recursion/latest/async_recursion/)
+
+
+## Lifetimes and borrowing
+
+- Mentioned the static lifetime above
+- Lifetime bounds on futures (`Future + '_`, etc.)
+- Borrowing across await points
+- I don't know, I'm sure there are more lifetime issues with async functions ...

src/part-guide/concurrency-primitives.md

@@ -0,0 +1,25 @@
+# Concurrency primitives
+
+- concurrent composition of futures
+  - c.f., sequential composition with await, composition of tasks with spawn
+  - concurrent/task behaviour
+  - behaviour on error
+- streams as alternative, forward ref
+- different versions in different runtimes/other crates
+  - focus on the Tokio versions
+
+## Join
+
+- Tokio/futures-rs join macro
+- c.f., joining tasks
+- join in futures-concurrency
+- FuturesUnordered
+  - like a dynamic version of join
+  - forward ref to stream
+
+## Race/select
+
+- Tokio select macro
+- cancellation issues
+- different behaviour of futures-rs version
+- race in futures-concurrency

src/part-guide/dtors.md

@@ -0,0 +1,42 @@
+# Destruction and clean-up
+
+- Object destruction and recap of Drop
+- General clean up requirements in software
+- Async issues
+  - Might want to do stuff async during clean up, e.g., send a final message
+  - Might need to clean up stuff which is still being used async-ly
+  - Might want to clean up when an async task completes or cancels and there is no way to catch that
+  - State of the runtime during clean-up phase (esp if we're panicking or whatever)
+  - No async Drop
+    - WIP
+  - forward ref to completion io topic
+
+## Cancellation
+
+- How it happens (recap of adv-async-await.md)
+  - drop a future
+  - cancellation token
+  - abort functions
+- What we can do about 'catching' cancellation
+  - logging or monitoring cancellation
+- How cancellation affects other futures tasks (forward ref to cancellation safety chapter, this should just be a heads-up)
+
+## Panicking and async
+
+- Propagation of panics across tasks (spawn result)
+- Panics leaving data inconsistent (tokio mutexes)
+- Calling async code when panicking (make sure you don't)
+
+## Patterns for clean-up
+
+- Avoid needing clean up (abort/restart)
+- Don't use async for cleanup and don't worry too much
+- async clean up method + dtor bomb (i.e., separate clean-up from destruction)
+- centralise/out-source clean-up in a separate task or thread or supervisor object/process
+
+## Why no async Drop (yet)
+
+- Note this is advanced section and not necessary to read
+- Why async Drop is hard
+- Possible solutions and there issues
+- Current status

src/part-guide/futures.md

@@ -0,0 +1,44 @@
+# Futures
+
+We've talked a lot about futures in the preceding chapters; they're a key part of Rust's async programming story! In this chapter we're going to get into some of the details of what futures are and how they work, and some libraries for working directly with futures.
+
+## The `Future` and `IntoFuture` traits
+
+- Future
+  - Output assoc type
+  - No real detail here, polling is in the next section, reference adv sections on Pin, executors/wakers
+- IntoFuture
+  - Usage - general, in await, async builder pattern (pros and cons in using)
+- Boxing futures, `Box<dyn Future>` and how it used to be common and necessary but mostly isn't now, except for recursion, etc.
+
+## Polling
+
+- what it is and who does it, Poll type
+  - ready is final state
+- how it connects with await
+- drop = cancel
+  - for futures and thus tasks
+  - implications for async programming in general
+  - reference to chapter on cancellation safety
+
+### Fusing
+
+## futures-rs crate
+
+- History and purpose
+  - see streams chapter
+  - helpers for writing executors or other low-level futures stuff
+    - pinning and boxing
+  - executor as a partial runtime (see alternate runtimes in reference)
+- TryFuture
+- convenience futures: pending, ready, ok/err, etc.
+- combinator functions on FutureExt
+- alternative to Tokio stuff
+  - functions
+  - IO traits
+
+## futures-concurrency crate
+
+https://docs.rs/futures-concurrency/latest/futures_concurrency/
+
+

src/part-guide/io.md

@@ -0,0 +1,59 @@
+# IO and issues with blocking
+
+## Blocking and non-blocking IO
+
+- High level view
+- How async IO fits with async concurrency
+- Why blocking IO is bad
+- forward ref to streams for streams/sinks
+
+## Read and Write
+
+- async Read and Write traits
+  - part of the runtime
+- how to use
+- specific implementations
+  - network vs disk
+    - tcp, udp
+    - file system is not really async, but io_uring (ref to that chapter)
+  - practical examples
+  - stdout, etc.
+  - pipe, fd, etc.
+
+
+## Memory management
+
+- Issues with buffer management and async IO
+- Different solutions and pros and cons
+  - zero-copy approach
+  - shared buffer approach
+- Utility crates to help with this, Bytes, etc.
+
+## Advanced topics on IO
+
+- buf read/write
+- Read + Write, split, join
+- copy
+- simplex and duplex
+- cancelation
+
+## The OS view of IO
+
+- Different kinds of IO and mechanisms, completion IO, reference to completion IO chapter in adv section
+  - different runtimes can faciliate this
+  - mio for low-level interface
+
+
+## Other blocking operations
+
+- Why this is bad
+- Long running CPU work
+  - Using Tokio for just CPU work: https://thenewstack.io/using-rustlangs-async-tokio-runtime-for-cpu-bound-tasks/
+- Solutions
+  - spawn blocking
+  - thread pool
+  - etc.
+- yielding to the runtime
+  - not the same as Rust's yield keyword
+  - await doesn't yield
+  - implicit yields in Tokio

src/part-guide/runtimes.md

@@ -0,0 +1,29 @@
+# Runtimes and runtime issues
+
+## Running async code
+
+- Explicit startup vs async main
+- tokio context concept
+- block_on
+- runtime as reflected in the code (Runtime, Handle)
+- runtime shutdown
+
+## Threads and tasks
+
+- default work stealing, multi-threaded
+  - revisit Send + 'static bounds
+- yield
+- spawn-local
+- spawn-blocking (recap), block-in-place
+- tokio-specific stuff on yielding to other threads, local vs global queues, etc
+
+## Configuration options
+
+- thread pool size
+- single threaded, thread per core etc.
+
+## Alternate runtimes
+
+- Why you'd want to use a different runtime or implement your own
+- What kind of variations exist in the high-level design
+- Forward ref to adv chapters

src/part-guide/streams.md

@@ -0,0 +1,43 @@
+# Async iterators (FKA streams)
+
+- Stream as an async iterator or as many futures
+- WIP
+  - current status
+  - futures and Tokio Stream traits
+  - nightly trait
+- lazy like sync iterators
+- pinning and streams (forward ref to pinning chapter)
+- fused streams
+
+## Consuming an async iterator
+
+- while let with async next
+- for_each, for_each_concurrent
+- collect
+- into_future, buffered
+
+## Stream combinators
+
+- Taking a future instead of a closure
+- Some example combinators
+- unordered variations
+
+## Implementing an async iterator
+
+- Implementing the trait
+- Practicalities and util functions
+- async_iter stream macro
+
+## Sinks
+
+- https://docs.rs/futures/latest/futures/sink/index.html
+
+## Future work
+
+- current status
+  - https://rust-lang.github.io/rfcs/2996-async-iterator.html
+- async next vs poll
+- async iteration syntax
+- (async) generators
+- lending iterators
+

src/part-guide/sync.md

@@ -0,0 +1,41 @@
+# Channels, locking, and synchronization
+
+note on runtime specificness of sync primitves
+
+Why we need async primitives rather than use the sync ones
+
+## Channels
+
+- basically same as the std ones, but await
+  - communicate between tasks (same thread or different)
+- one shot
+- mpsc
+- other channels
+- bounded and unbounded channels
+
+## Locks
+
+- async Mutex
+  - c.f., std::Mutex - can be held across await points (borrowing the mutex in the guard, guard is Send, scheduler-aware? or just because lock is async?), lock is async (will not block the thread waiting for lock to be available)
+    - even a clippy lint for holding the guard across await (https://rust-lang.github.io/rust-clippy/master/index.html#await_holding_lock)
+  - more expensive because it can be held across await
+    - use std::Mutex if you can
+      - can use try_lock or mutex is expected to not be under contention
+  - lock is not magically dropped when yield (that's kind of the point of a lock!)
+  - deadlock by holding mutex over await
+    - tasks deadlocked, but other tasks can make progress so might not look like a deadlock in process stats/tools/OS
+    - usual advice - limit scope, minimise locks, order locks, prefer alternatives
+  - no mutex poisoning
+  - lock_owned
+  - blocking_lock
+    - cannot use in async
+  - applies to other locks (should the above be moved before discussion of mutex specifically? Probably yes)
+- RWLock
+- Semaphore
+- yielding
+
+## Other synchronization primitives
+
+- notify, barrier
+- OnceCell
+- atomics