UIP-0103: Persistent Nock Caching + Loom Migration Framework + Pointer Compression Migration Refactor #508

matthew-levan · 2023-09-01T20:43:39Z

Implements UIP-0103, establishes a pattern for implementing loom migrations, and refactors the pointer compression as a compile-time feature instead of a runtime one.

See #496 for original work on UIP-0103.

TODO:

Loom migration
Affordance for future specialized caches in u3a_road struct
PRs for use cases ready in Arvo
- ames: persistently cache ++jim and remove ++dedup-message urbit#6770
- arvo: persistently cache scry type checks urbit#6773
- ~~Ford Lightning~~
Manage new persistent state like we manage the rest of our persistent state (|mass breaks before we do this)
Update command-line args for cache sizes
Determine why bazel build //pkg/vere:test-fake-ship seems to hang indefinitely (I ran it for ~16h on my own dedicated box and it still didn't finish)

Tested working in all vanes I tested (Iris, etc. are a little more cumbersome):

matthew-levan · 2023-09-11T15:55:23Z

Notes from @barter-simsum:

Consider defining a macro to generate versioned symbols, and rename u3j_vX..., etc. as _mig_vX...

Example of a versioned macro:

#define VERSIONED(x) _mig_v ## x
VERSIONED(_cj_free_hank) (params) { defn }

pkg/noun/allocate.h

pkg/noun/allocate.c

pkg/noun/jets.c

pkg/noun/jets.h

pkg/noun/nock.h

pkg/noun/v1/jets.h

pkg/noun/v1/allocate.h

pkg/noun/v1/nock.c

pkg/noun/v1/nock.h

pkg/noun/v3/manage.c

pkg/noun/allocate.h

pkg/noun/manage.c

pkg/noun/allocate.h

pkg/noun/manage.h

pkg/noun/options.h

pkg/noun/v2/manage.c

pkg/noun/v3/manage.c

pkg/noun/v3/vortex.h

pkg/noun/v2/nock.h

This PR adjusts the serf's memory-pressure thresholds, adds state to track threshold events and avoid thrashing ("schmitt trigger"), and refactors post-event operation to use a bitmap and track operations more granularly. These changes are useful on there own, and necessary for adding memory-pressure-based trimming to #508.

joemfb

this will do

This PR dedicates slot 0 to cells (`u3a_minimum`), and aligns subsequent slots on power-of-two boundaries. Previously, slot 0 was unused, and subsequent slots were aligned *near* power-of-two boundaries. For example: | words | old slot | new slot | | --- | - | - | | 6 | 1 | 0 | | 31 | 3 | 2 | | 61 | 4 | 3 | | 62 | 4 | 3 | | 63 | 4 | 3 | | 121 | 5 | 4 | | 122 | 5 | 4 | | 123 | 5 | 4 | | 124 | 5 | 4 | | 125 | 5 | 4 | | 126 | 5 | 4 | | 127 | 5 | 4 | The first issue dates to urbit/urbit#987. Slot 0 was reserved for sizes less than 8, which meant that 7-word allocations on the home road were incredibly slow (as they traversed a free list full of 6 word allocations). The second issue has always been the case in this allocator, due to the way that the size was rounded up on each iteration of the "power-of-two" sizing loop. This PR does not address the infamous size-bumping logic in the allocator: https://github.com/urbit/vere/blob/9bdc1af00f6650e27ac66793c6781d933dd30dfe/pkg/noun/allocate.c#L453-L467 I'm been tempted to disable that behavior on the home-road (to further reduce fragmentation), but I'm not convinced that won't still run into pathological performance issues. The best approach might be to start searching the "proper" free list, but bound the number of iterations before bumping. Either change can be made at any time, without migrations. This PR does requires a migration to move free space into the now-appropriate free-list. It includes a trivial, always on migration, which should be refactored into #508 when appropriate. This PR helps somewhat with urbit/urbit#6805, as it reduces home-road fragmentation.

This PR builds on #546, extending it's more-granular state management to the persistent memoization cache introduced in #508

matthew-levan added the feature New feature or feature request label Sep 1, 2023

matthew-levan force-pushed the msl/loom-migration-refactor branch from 3901404 to 0bbe610 Compare September 6, 2023 19:21

matthew-levan changed the title ~~Loom migration "framework" (with pointer compression migration refactor)~~ Loom migration framework (with pointer compression migration refactor) Sep 6, 2023

matthew-levan force-pushed the msl/loom-migration-refactor branch 2 times, most recently from f35cd83 to 8570654 Compare September 8, 2023 19:39

matthew-levan marked this pull request as ready for review September 11, 2023 15:13

matthew-levan requested a review from a team as a code owner September 11, 2023 15:13

matthew-levan force-pushed the msl/loom-migration-refactor branch 2 times, most recently from 69f75b9 to 4cc9d1e Compare September 15, 2023 16:57