feat(daemon): store endo bootstrap and default host formulas #2089

kumavis · 2024-02-17T22:53:53Z

Factors out endo configuration ( eg default host formulaIdentifier ) from newly named daemonCore.
Introduces incarnateXyz methods to be used when creating a new thing for the first time.
Persists all formulas and requires them to be incarnated before they can be reified via provideValueForFormulaId
Adds Handles as a generic pointer to another formula, where the pointer is exposed on the internal facet only. This is currently used to give a guest a reference to the host without exposing host endowments. mail uses provideControllerForFormulaIdentifierAndResolveHandle to resolve the handle.

Updates: #1993
Closes: #2087

…ResolveHandle

kumavis · 2024-02-18T23:26:04Z

once witnessed an unreproduceable EPIPE in CI 🤔
https://github.com/endojs/endo/actions/runs/7952406406/job/21706961986?pr=2089

kumavis · 2024-02-19T05:46:37Z

EPIPE seen on windows CI https://github.com/endojs/endo/actions/runs/7955058144/job/21713421990?pr=2089

rekmarks

Looks promising!

packages/daemon/src/daemon-node-powers.js

packages/daemon/src/types.d.ts

packages/daemon/src/mail.js

packages/daemon/src/guest.js

rekmarks · 2024-02-20T00:23:14Z

packages/daemon/src/types.d.ts

+  host: string;
+  leastAuthority: string;
+  webPageJs?: string;


It would be neat to document what the string properties on all of our formulas are supposed to be. Names? Identifiers if the real thing? Identifiers of handles?

how to make subtypes of string? so you can't give it a string unless you cast it to a formulaId? or not a good idea?

Ah, I don't think it's worth doing anything with template literal types here (especially in light of #2088). I just meant documenting what they're supposed to be on the interfaces would be nice.

I think we're OK for now since that information is there when the formulas are instantiated, so I won't block on it.

packages/daemon/src/daemon.js

rekmarks · 2024-02-20T01:19:33Z

One more question: what's the difference between incarnating something and reifying it? We should perhaps document this.

… isNewlyCreated

kumavis · 2024-02-20T02:43:27Z

One more question: what's the difference between incarnating something and reifying it? We should perhaps document this.

this PR introduces "incarnate" as a verb to mean to create a new thing for the first time. it follows that we can use "reincarnate" to reify things. fun. But would not be surprised if we land on different language.

rekmarks

LGTM!

Pushed up some minor changes in 77b1e68. Fixed broken web-bundle formula numbers in ba4c257.

kumavis · 2024-02-20T17:43:56Z

thanks for the improvements/

kriskowal · 2024-02-21T00:40:59Z

packages/daemon/src/daemon.js

+        assertPetName,
+      );
+      return { external, internal: undefined };
+    } else if (formula.type === 'pet-inspector') {


We could probably turn this would-be-switch block into a JavaScript map from type to behavior now.

Synchronizes the host's `evaluate()` method by delegating all incarnations to the daemon's `incarnateEval()`. Also synchronizes `incarnateLookup()`, and removes `provideLookupFormula()` from the mailbox, which was only a thin wrapper anyway. The path taken introduces new incarnation semantics and design constraints on the daemon and its dependents. #2089 introduced the notion of "incarnating" values. The `incarnateX() methods are responsible for creating formula JSON objects and calling `provideValueForNumberedFormula` to reify their values for the first time. Synchronizing dependencies between `incarnateX` methods is feasible by means of a lock, but difficulties arise when formulas are incarnated outside the daemon, such as in the host. Coordinating a lock between two different modules will increase the risks of datalocks at runtime. To avoid this, the implementation delegates all incarnations necessary for `evaluate()` to its dependent `incarnateEval()`. In essence, it is the consumer's responsibility to specify all necessary incarnations to the relevant incarnation method, which is then responsible for carrying them out. Yet, the story is complicated by pet names being mostly abstracted away from the daemon. For example, pet names must be associated with their respective formula identifiers the moment that those identifiers are observable to the consumer. As part of this process, the pet store must write the name to formula id mapping to disk. To handle sych asynchronous side effects, the implementation introduces a notion of "hooks" to `incarnateEval()`, with the intention of spreading this to other incarnation methods as necessary. These hooks receive as an argument all formula identifiers created by the incarnation, and are executed under the formula graph synchronization lock. This will surface IO errors to the consumer, and help us uphold the principle of "death before confusion".

Synchronizes the host's `evaluate()` method by delegating all incarnations to the daemon's `incarnateEval()`. Also synchronizes `incarnateLookup()`, and removes `provideLookupFormula()` from the mailbox, which was only a thin wrapper anyway. The path taken introduces new incarnation semantics and design constraints on the daemon and its dependents. #2089 introduced the notion of "incarnating" values. The `incarnateX() methods are responsible for creating formula JSON objects and calling `provideValueForNumberedFormula` to reify their values for the first time. Synchronizing dependencies between `incarnateX` methods is feasible by means of a lock, but difficulties arise when formulas are incarnated outside the daemon, such as in the host. Coordinating a lock between two different modules will increase the risks of datalocks at runtime. To avoid datalocks, the implementation delegates all incarnations necessary for `evaluate()` to its dependent `incarnateEval()`. In essence, it is the consumer's responsibility to specify all necessary incarnations to the relevant incarnation method, which is then responsible for carrying them out. Yet, the story is complicated by pet names being mostly abstracted away from the daemon. For example, pet names must be associated with their respective formula identifiers the moment that those identifiers are observable to the consumer. As part of this process, the pet store must write the name to formula id mapping to disk. To handle sych asynchronous side effects, the implementation introduces a notion of "hooks" to `incarnateEval()`, with the intention of spreading this to other incarnation methods as necessary. These hooks receive as an argument all formula identifiers created by the incarnation, and are executed under the formula graph synchronization lock. This will surface IO errors to the consumer, and help us uphold the principle of "death before confusion". Also of note, `provideValueForNumberedFormula` has been modified such that the formula is written to disk _after_ the controller has been constructed. This is critical in order to serialize in-memory formula graph mutations. Since it's possible for incarnation to fail after the controller has been created, we should consider adding cleanup callbacks to incarnation hooks.

Synchronizes the host's `evaluate()` method by delegating all incarnations to the daemon via `incarnateEval()`. The latter is responsible for incarnating its dependents as necessary, and for the generation of their formula numbers. To facilitate this, the synchronous methods `incarnateLookupSync()` and `incarnateWorkerSync()` have been added. These methods synchronously mutate the formula graph, and return promises that resolve when the formulas have been written to disk. The result is that the formula graph is mutated within a single turn of the event loop. To achieve this, the implementation introduces new constraints on the daemon and its dependents. #2089 introduced the notion of "incarnating" values. By the current definition, incarnating a value consists of the following steps: 1. Collecting dependencies (async) a. Generating requisite formula numbers (async) - We use the asynchronous signature of `crypto.randomBytes` to do this for performance reasons. b. Incarnating any dependent values (recursion!) 2. Updating the in-memory formula graph (sync) 3. Writing the resulting formula to disk (async) 4. Reifiying the resulting value (async) In order to make formula graph mutations mutually exclusive, we introduce a "formula graph mutex" under which step 1 must be performed. This mutex is currently only used by `incarnateEval()`, and must be expanded to its sibling methods in the future. `incarnateEval()` also introduces the notion of "incarnation hooks" to the codebase. Originators of incarnations that are exogenous to the daemon may themselves have asynchronous work perform. For example, `petName -> formulaIdentifier` mappings are the responsibility of the host and its pet store, and pet names must be associated with their respective formula identifiers the moment that those identifiers are observable to the consumer. To handle sych asynchronous side effects, the implementation introduces a notion of "hooks" to `incarnateEval()`, with the intention of spreading this to other incarnation methods as necessary. These hooks receive as an argument all formula identifiers created by the incarnation, and are executed under the formula graph mutex. This will surface IO errors to the consumer, and help us uphold the principle of "death before confusion". Also of note, `provideValueForNumberedFormula` has been modified such that the formula is written to disk _after_ the controller has been constructed. This is critical in order to synchronize formula graph mutations. Finally, it appears that the implementation incidentally fixed #2021. We may still wish to adopt the more robust solution proposed in that issue.

kumavis added 2 commits February 17, 2024 12:37

feat(daemon): store endo formula upon initialization

dd0d64d

refactor(daemon): breakout makeDaemonCore from makeEndoBootstrap

e90e4fa

kumavis force-pushed the kumavis-daemon-handle branch from 30bbfbc to 9519f27 Compare February 18, 2024 04:48

kumavis added 3 commits February 17, 2024 22:08

feat(daemon): must use incarnateHost to make a new host

89d9418

feat(daemon): must use incarnateBundler to make the web-page-js bundler

1f694c7

refactor(daemon): incarnate endo daemon dependencies before endo formula

d598be7

kumavis force-pushed the kumavis-daemon-handle branch from 60046db to d598be7 Compare February 18, 2024 08:12

feat(daemon): add handle and provideControllerForFormulaIdentifierAnd…

f92d751

…ResolveHandle

kumavis force-pushed the kumavis-daemon-handle branch 3 times, most recently from 0340e35 to 252853e Compare February 18, 2024 23:07

refactor(daemon): first time init via incarnateEndoBootstrap

c5a001c

kumavis force-pushed the kumavis-daemon-handle branch from 252853e to c5a001c Compare February 18, 2024 23:18

kumavis added 3 commits February 18, 2024 13:27

feat(daemon): add incarnateGuest

e82a7b7

feat(daemon): add incarnateLeastAuthority

073d46d

feat(daemon): add incarnateEval

0bd689d

kumavis force-pushed the kumavis-daemon-handle branch from 19a7614 to 0bd689d Compare February 19, 2024 00:34

kumavis added 2 commits February 18, 2024 18:09

feat(daemon): add incarnateReadableBlob

628f903

feat(daemon): add incarnateUnconfined

b391c25

kumavis force-pushed the kumavis-daemon-handle branch from c00b093 to b391c25 Compare February 19, 2024 04:10

kumavis added 3 commits February 18, 2024 18:27

feat(daemon): add incarnateBundle

41fa16b

feat(daemon): add incarnateWebBundle

bbcdc11

feat(daemon): add incarnateWorker

729d4aa

kumavis force-pushed the kumavis-daemon-handle branch from d110daf to 729d4aa Compare February 19, 2024 05:17

feat(daemon): add incarnatePetInspector

a108e7b

kumavis marked this pull request as ready for review February 19, 2024 07:56

kumavis added 2 commits February 19, 2024 05:26

feat(daemon): add incarnatePetStore

8cece42

feat(daemon): add incarnateLookup

9fbbfb3

kumavis added 2 commits February 19, 2024 05:45

fix(daemon): correct getWebPageBundlerFormula args

21b5750

feat(daemon): expose all incarnate methods on daemonCore

0c6daf5

kumavis requested review from rekmarks and kriskowal February 19, 2024 23:16

rekmarks requested changes Feb 20, 2024

View reviewed changes

kumavis added 3 commits February 19, 2024 15:28

refactor(daemon): DaemonicPersistencePowers provideRootNonce includes…

075d2be

… isNewlyCreated

fix(daemon): rename EndoBootstrap type to FarEndoBootstrap

d643f0c

doc(daemon): add doc for Handle and InternalHandle in types.d.ts

169c26c

kumavis force-pushed the kumavis-daemon-handle branch from 35ece3f to 169c26c Compare February 20, 2024 02:09

kumavis added 2 commits February 19, 2024 16:17

test(daemon): ensure guest cannot access HOST methods

bdcb3ba

doc(daemon): improve doc introducing makeDaemonCore internals

1ae8717

refactor(daemon): Update handle documentation and types

77b1e68

rekmarks approved these changes Feb 20, 2024

View reviewed changes

fix(daemon): Restore 128-bit formula numbers for web bundles

e5d7082

rekmarks force-pushed the kumavis-daemon-handle branch from ba4c257 to e5d7082 Compare February 20, 2024 08:27

kumavis merged commit e9f7049 into master Feb 20, 2024
14 checks passed

kumavis deleted the kumavis-daemon-handle branch February 20, 2024 17:45

kriskowal reviewed Feb 21, 2024

View reviewed changes

rekmarks mentioned this pull request Feb 22, 2024

refactor(daemon): Synchronize host evaluate method #2092

Merged

rekmarks mentioned this pull request Feb 22, 2024

daemon: Instantiate one lookup formula for each segment in a path #2098

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(daemon): store endo bootstrap and default host formulas #2089

feat(daemon): store endo bootstrap and default host formulas #2089

kumavis commented Feb 17, 2024 •

edited by rekmarks

Loading

kumavis commented Feb 18, 2024 •

edited

Loading

kumavis commented Feb 19, 2024

rekmarks left a comment

rekmarks Feb 20, 2024

kumavis Feb 20, 2024

rekmarks Feb 20, 2024

rekmarks commented Feb 20, 2024

kumavis commented Feb 20, 2024

rekmarks left a comment •

edited

Loading

kumavis commented Feb 20, 2024

kriskowal Feb 21, 2024

feat(daemon): store endo bootstrap and default host formulas #2089

feat(daemon): store endo bootstrap and default host formulas #2089

Conversation

kumavis commented Feb 17, 2024 • edited by rekmarks Loading

kumavis commented Feb 18, 2024 • edited Loading

kumavis commented Feb 19, 2024

rekmarks left a comment

Choose a reason for hiding this comment

rekmarks Feb 20, 2024

Choose a reason for hiding this comment

kumavis Feb 20, 2024

Choose a reason for hiding this comment

rekmarks Feb 20, 2024

Choose a reason for hiding this comment

rekmarks commented Feb 20, 2024

kumavis commented Feb 20, 2024

rekmarks left a comment • edited Loading

Choose a reason for hiding this comment

kumavis commented Feb 20, 2024

kriskowal Feb 21, 2024

Choose a reason for hiding this comment

kumavis commented Feb 17, 2024 •

edited by rekmarks

Loading

kumavis commented Feb 18, 2024 •

edited

Loading

rekmarks left a comment •

edited

Loading