Unlocked deps

[macovedj]

This PR servers as an initial description of what the desired behavior of an unlocked-dep would include. As exemplified in the PR, most language toolchains would probably use it in combination with nested interfaces, which is currently in the process of being implemented.

The primary motivation behind this syntax is enabling a way for registry tooling concerned with package resolution to resolve implementations of components (rather than wit). This should hopefully enable workflows where users can depend on specific components, rather than interfaces whose implementations need to be specified at a later time.

[lukewagner]

Thanks for starting this! I have some suggested changes to the examples, WDYT:

[macovedj]

Appreciate the review! Makes sense to me.

[alexcrichton]

Is it intended to support foo:bar here exactly? Or is a /interface required as well?

[lukewagner]

I was thinking that, when /interface is absent, the expectation could be that a registry is used to resolve ns:package to a component and the interface is the exports of that component (which have no name).

More hypothetically, if we get unnamed interfaces/worlds in WIT, /interface could also be absent without relying on a registry by using a nested package of the form:

package foo:bar {
  interface { ... }
}
world w {
  unlocked-dep foo:bar@{...}
}

WDYT?

[macovedj]

But perhaps this is a good opportunity to make clear that the optional /interface is possible as well, and maybe talk a bit about in which cases toolchains would create an unlocked-dep with and without an interface projection? If we wanted to scope it in, folks could specify an interface when they add a dep and tools could only grab interfaces they need from the registry. And also the lock or bundle command could shake out unused interfaces in cases as well. May be more than we want for the initial discussion, but I was thinking that folks may find it confusing to only see a component foo:bar when they're accustomed to depending on interfaces at the moment.

[lukewagner]

Fundamentally, an unlocked-dep (in the component importname) names a component via package name and so, for a simple illustrative example like this, I think makes sense to similarly start with unlocked-dep naming a package. (Yes, it's different that regular interface imports, but being different is the point.) The /interface only shows up after inlining registry contents and is only necessary due to current expressive limitations of WIT; ideally it wouldn't ever be necessary.

[alexcrichton]

Could this spec out the interface { ... } change as well in that case? That's not currently implemented or sketched out here, so I think that should be included too if that's the intention. (also could this update the ebnf for unlocked-dep in worlds too?)

[lukewagner]

I'd be up for that; it would simplify things.

[macovedj]

Great I should be able to add some of these updates soon.

[lukewagner]

As an alternative to adding unnamed interfaces/worlds (which open up another set of design questions), we could instead add a bit of also-useful syntax that allows the WIT to capture precisely how the range query was resolved:

package gh:sqlite@1.1.1 {
  interface exports { ... }
}
world w {
  import dependency gh:sqlite@{>=1.0.0} = gh:sqlite/[email protected];
}

This = <pkgname> right-hand-side of the import dependency is necessary for advanced cases where the WIT needs to nest multiple versions of the same package such that the <pkgnamequery> alone would be ambiguous. Importantly for our purposes here, though, none of the name/version info in the = <pkgname> shows up in the import so our world w above just contains (import "unlocked-dep=<gh:sqlite@{>=1.0.0}>" (instance ...)) (no /exports, no 1.1.1), and we didn't need anonymous interfaces to achieve that.

[James-Mart]

Not sure if this is the best place to add my opinion on this locked/unlocked syntax. Let me know if there would be a better place to express this.

As a user of the component-model, as opposed to a core developer such as yourselves, the meaning of the locked-dep and unlocked-dep keywords are not obvious to me. After reading the proposed updates in this PR to Wit.md (and the Explainer.md), I understand that unlocked-dep indicates a dependency "on a component implementation (or a semver range of implementations), rather than on an abstract WIT interface (with an unspecified implementation)", but to me that does not explain why the keyword "unlocked-dep" was chosen to express this.

Seems to me that if the WIT author is concerned with expressing the simple concept of a dependency on a specific component vs on any component that conforms to the specified interface, then perhaps wit could have more self-explanatory keywords like specified-dep and specified-dep-range?

In fact, couldn't it use a single keyword? Something like:

depname       ::= 'specified-dep=<' <pkgnamequery> | <pkgname> '>' ( ',' <hashname> )?

If needed, it seems a wit parser could convert the above syntax into the a corresponding locked-dep / unlocked-dep syntax for wat representation.

Just my two-cents!

Thanks,
James

[James-Mart]

Related to my comment on this PR, but an "unlocked component" really doesn't seem self-explanatory to me. Language like, "coupled component" or "tightly coupled component" seem to more readily express the fact that the component depends on implementations rather than interfaces.

[macovedj]

I think this is a great place for you to voice such an opinion @James-Mart, really appreciate the feedback!

Seems to me that if the WIT author is concerned with expressing the simple concept of a dependency on a specific component vs on any component that conforms to the specified interface, then perhaps wit could have more self->explanatory keywords like specified-dep and specified-dep-range?

Something that may impact your opinion... At the moment we don't really anticipate that developers would ever author this syntax in WIT themselves, or at least not often. More commonly, toolchains would deduce that this WIT should be used for bindings generation based on a dependency specification in a package file.

As for the syntax suggestions, I don't feel super strongly one way or the other immediately. It's valuable to hear that some of the verbiage/syntax doesn't feel intuitive to you, but I also think we have yet to fully document the entire lifecycle of an unlocked dep/unlocked component. I was having a bit of difficulty deciding how much of this information belongs in this PR vs in other places, but I'll try and add some context that helps provide some clarity around some of the points you're raising.

[lukewagner]

@James-Mart Even if mostly folks won't manually write their component dependencies in WIT often, I'm definitely happy to discuss what is the most intuitive syntax/naming that we can find for WIT (which I agree can be distinct from what goes into the raw WAT) since I think folks will at least occasionally read synthesized WITs that use this syntax. So thanks for bringing this up!

One observation is that both unlocked-dep and your proposed specified-dep/speified-dep-range share the (abbreviation of the) word "dependency". What if we used just the word "dependency" and inferred "locked" vs. "unlocked"/"range" from the syntax after the @. E.g.:

world w {
  dependency ns:foo@{>=1.0.0}; // => unlocked-dep=<ns:foo@{>=1.0.0}>
  dependency ns:[email protected];     // => locked-dep=<ns:[email protected]>
}

(We'd also need to invent some syntax for where to include the integrity= content hash for a locked-dep.)

Separately, I've wondered whether there's value in having an explicit import prefix:

world w {
  import dependency ns:foo@{>=1.0.0};

Syntactically, there's no need for import (dependencies can only be imported, not exported), but I wonder if there's value in having it for regularity dependencies are in fact imported? Happy to hear other opinions on this.

As a last detail, I think that dependencys would always emits an import with an instance type. Eventually (I don't think we need to add or choose it in this PR), we'll also need a syntax that emits a component type (which tends to show up when you lock your dependencies, but, at the component-model level, is orthogonal). Observing that a component-typed import allows you to create private child instances (not shared with any other component) whereas an instance-typed import implies/expects that the same imported instance can be shared with other components, maybe private dependency could imply component type? Whatever the precise syntax, this is the rare case which I expect folks won't see often (even in synthesized WIT), since component imports mostly only show up at the end of building an application (out of unlocked components).

[James-Mart]

@macovedj

At the moment we don't really anticipate that developers would ever author this syntax in WIT themselves, or at least not often.

Ah, that does make sense for general use. Although I think in the context of my use-case for component development this may be the mechanism that developers are instructed to use pretty regularly, since they are building components that are always dynamically composed, and there can be multiple implementations of each interface, so developers need some way to indicate to the runtime which instance should be linked.

@lukewagner

What if we used just the word "dependency"

What about just instance? Seems potentially better than dependency at conveying the idea that it's specifically a dependency on an instance, rather than an interface. But it probably only makes sense with explicit import: import instance ns:[email protected];. If import is optional, then maybe dependency is clearer on the whole.

and inferred "locked" vs. "unlocked"/"range" from the syntax after the @

Definitely +1 for this

I wonder if there's value in having [import] for regularity dependencies are in fact imported?

Without the explicit import, I think a reader is more likely to assume that dependency is some kind of top-level directive (like import and export), rather than a specialization of import. So I like the explicit import, If not required I think it would be helpful if it were at least allowed.

[lukewagner]

What about just instance? Seems potentially better than dependency at conveying the idea that it's specifically a dependency on an instance, rather than an interface.

That's a good thought, and it makes technical sense if you're familiar with how instance definitions and types work in the C-M, but in my experience saying "instance" almost always ends up confusing folks if they're not already intimately familiar with the C-M, so I've learned to avoid leaning on it when trying to be accessible.

The nice thing about the word "dependency" is that, from what I've seen, the word "dependency" is used by package managers and their associated build-config files (e.g., npm/package.json, cargo/Cargo.toml, etc) to exclusively refer to implementations. And in addition to being able to import these dependencies, you can also import things directly from the host that are not in the dependencies (e.g., in Node, import { readFile } from 'node:fs';), suggesting that import is the more general "pull in functionality" word whereas dependency is the "pull in a particular implementation" word.

Just to see an example of dependency imports mixed in with other imports:

world w {
  import wasi:keyvalue/store;
  import wasi:http/handler;
  import dependency gh:sqlite@{>=1.0.0};
}

IMO, that seems decently unambiguous (and an improvement from the original proposed syntax). But that's just from my limited perspective; happy to discuss more!