west init: CLI argument to automatically setup new workspace from template

[jamie-bes]

There are three basic types of Zephyr applications: Zephyr repository applications, workspace applications, and freestanding applications.

The default behavior of west init without any arguments effectively creates a Zephyr repository application, as upstream Zephyr becomes the manifest repository. This is nice for building the samples included in the Zephyr source tree, but creating a new application within this structure typically requires forking Zephyr. The workspace application structure is often better for developing new applications, as the repo for your application only needs to contain the application itself while Zephyr gets pulled in as a module. But the workflow for creating a workspace application is a bit more verbose, as you need to provide an already-existing manifest repository with a west.yml file and pass either its URL (with -m) or a local file path (with -l) to West.

My proposal is to add a new argument that will automatically create a template manifest repository with west.yml and other required files, such that it can immediately be used as a workspace application. The idea comes from Rust's cargo init, which automatically initializes a new Git repo with Cargo.toml, Cargo.lock, .gitignore and a src/main.rs with a basic "Hello World" function. This makes starting a new application extremely easy, especially for beginners, so I think it'd be a nice feature to replicate in West.

I think west init --template or west init -t for short would be a decent name, though I'm open to other ideas. Since this probably couldn't be used in combination with -m and --mr due to the manifest repository being created from scratch, it'd be handy to have additional arguments to specify which version of Zephyr to include in west.yml (maybe something like west init -t --base https://github.com/nrfconnect/sdk-nrf and west init -t --br v3.3.0).

As for what should be included in the template, I'd recommend this:

src/
├─ main.c
.gitignore
CMakeLists.txt
prj.conf
west.yml

...where .gitignore ignores the build directory, west.yml imports the selected base (defaulting to https://github.com/zephyrproject-rtos/zephyr) and nothing else, and the remaining files mimic the contents of the Hello World sample. I could also see an argument for using https://github.com/zephyrproject-rtos/example-application as the template, though it's a fairly maximalist template and provides more than the average user needs. So restricting the template to the bare minimum needed to run "Hello World" makes more sense to me.

If it helps, I'd be happy to put together a PR for this feature, though I created this issue to get some feedback on the idea first.

[marc-hb]

it'd be handy to have additional arguments to specify which version of Zephyr to include in west.yml (maybe something like west init -t --base https://github.com/nrfconnect/sdk-nrf

This template location argument should not be just "handy" but required and it should be designed ready to support projects that don't use Zephyr at all. I know the location of the west documentation makes that confusing but west is actually a tool independent of Zephyr and we want west to become more and more independent from Zephyr:

• Support use cases other than Zephyr RTOS #246

From #408 (comment)

We'd like to solve these types of problems in general by divorcing zephyr base setting from west itself,

• Non zephyr projects #261 (comment)
• Building outside of .west workspace with ZEPHYR_BASE not properly documented? zephyr#33521
• etc.

A bare west init command defaults to Zephyr and that can't be changed for obvious backwards compatibility reasons but that should be the exception not the rule. Other, newer options should require a Zephyr-specific parameter/URL

EDIT: this of course means most of the template details should live in the Zephyr repo, not in the west repo.

That caveat aside, the idea sounds good to me.

If it helps, I'd be happy to put together a PR for this feature, though I created this issue to get some feedback on the idea first.

Indeed! Thanks for the offer, much appreciated.

But please wait until @mbolivar-nordic / @mbolivar is back from vacation, I would hate you to write some code only to hear that it should be done but in a totally different way that he already had in his head for some time or even as a prototype in some branch.

[jamie-bes]

Ah I see - I wasn't aware of the efforts to make west more independent from Zephyr, so my proposal was pretty Zephyr-specific. I can try to generalize here.

This template location argument should not be just "handy" but required and it should be designed ready to support projects that don't use Zephyr at all.

A bare west init command defaults to Zephyr and that can't be changed for obvious backwards compatibility reasons but that should be the exception not the rule. Other, newer options should require a Zephyr-specific parameter/URL

I hadn't intended --base as a template location argument, but rather an override for whatever default imports the template has in its west.yml (eg. for Zephyr, overriding importing upstream Zephyr with a fork like NCS). So I considered it a nice-to-have but not strictly necessary as the user could just modify west.yml manually.

But if we want to explicitly specify what template to use (rather than defaulting to anything Zephyr-specific), we could add that as a required argument to --template. Using the URL of the template wouldn't have much value, as it would just be equivalent to west init -m https://github.com/zephyrproject-rtos/example-application -- the convenience of this proposal really comes from not needing to figure out what URL to use. But something like west init --template zephyr (shorthand west init -t zephyr) could work really well, where the zephyr template isn't built in to West itself but gets installed into some local directory of templates. And then if some other platform XYZ is also using West, a user could install a template for XYZ and be able to run west init -t xyz, without the West repo needing to have any implementation details specific to XYZ.

In #261, @mbolivar-nordic mentioned the idea of deprecating the bare west init that defaults to Zephyr.

We could similarly deprecate this slowly by emitting a warning in west init if -m is not given, and eventually make it a required argument over a period of a few years.

And for Zephyr users, I think west init -t zephyr could actually become the de-facto replacement for bare west init, providing the extra convenience of setting up a template "Hello World" app and being less verbose than west init -m https://github.com/zephyrproject-rtos/zephyr.

But please wait until @mbolivar-nordic / @mbolivar is back from vacation, I would hate you to write some code only to hear that it should be done but in a totally different way that he already had in his head for some time or even as a prototype in some branch.

Absolutely, no problem!

[marc-hb]

I don't know about you specifically but there's one thing I'm afraid a lot of people don't realize[*]: when they run west build, none of the code that runs is actually part of west. It all comes from https://github.com/zephyrproject-rtos/zephyr/tree/main/scripts/west_commands

[*] because they don't care, and that's OK.

[jamie-bes]

I see that @mbolivar / @mbolivar-ampere is back now - what are your thoughts on the idea of being able to install workspace templates and initialize them with west init -t <template_name>?

[mbolivar-ampere]

The general idea seems like a great way to save people time. I'd like to know more about how the "local directory of templates" should get set up and maintained and how the namespace will be controlled, though. Naively I too thought "this should be a URL", but I could easily be convinced otherwise with some more details about the ergonomics of that local directory!

[jamie-bes]

There's a lot of different ways that mapping between a name and a template could be handled. A package manager with a universal registry and controlled namespacing would likely just use some convention on top of their existing package namespacing scheme (like what Yarn does with yarn create and packages prefixed with create-).

But for West, I think the most logical approach is to just use a West workspace, since west.yml already handles assigning project names to repository URLs. The flow could look something like this:

1. An install script for West (either setup.py or something invoked by it) would do the following:
   • Initialize a new workspace in $HOME/.west_templates with an empty west.yml.
   • Run a command like west config --global templates.path $HOME/.west_templates to set West to use that workspace for looking up templates.
2. When installing Zephyr (or any other framework that uses West), their installation guide could optionally recommend any relevant template repositories and give an example snippet of what the user should copy into their $HOME/.west_templates/west.yml to "install" them.
3. Running west init -t my_template would look for a project named my_template in $HOME/.west_templates/west.yml and use that as the manifest repository to initialize a new workspace (which could just call west init -m <template_url> -mr <template_revision> under the hood).

Note that this flow doesn't actually run west update in the templates workspace -- it's just pulling info from its west.yml. But I could imagine another approach that does use west update and uses the cloned repository contents as a cache of the templates so they don't have to be downloaded every time you run west init -t <template_name>.

[marc-hb]

I'd like to know more about how the "local directory of templates" should get set up and maintained and how the namespace will be controlled, though.

Very important question indeed.

And for Zephyr users, I think west init -t zephyr could actually become the de-facto replacement for bare west init, providing the extra convenience of setting up a template "Hello World" app and being less verbose than west init -m https://github.com/zephyrproject-rtos/zephyr.

I like this. You could also extend it to west init -t zephyr/some_advanced_sample where west init -t zephyr is a shortcut for west init -t zephyr/default (similar to /index.html, you get the idea).

But for West, I think the most logical approach is to just use a West workspace, since west.yml already handles assigning project names to repository URLs. [...]
Note that this flow doesn't actually run west update in the templates workspace -- it's just pulling info from its west.yml.

Recursion is always sexy but I'm not convinced using west to manage a "registry" of templates brings a lot of value, it feels overkill to me. I mean you don't need a tool like west to implement a basic dictionary/map. Or rather, it feels premature: an implementation detail that should not taint the user interface anyway and that can be decided later.

For any registry like this I think the critical design decisions that must be right the first time (cause they're super hard to change later) are:

• how many layers of indirections
• how many "repositories" (typically: git repos but not necessarily)
• who has the authority (and the duty) to add/update/delete the template structure.

The hosting, download and caching technologies can be figured out later, maybe even enriched/diversified long after the first release.

It should always be possible to override from the default locations using some "long" arguments with full URLs (e.g. --registry http://myserver/myregistry or --template file:///home/me/mytemplate) but I understand that most users should not have to enter any URL; that's an important part of the desired feature

I think the 1st, "registry" level should be hosted and owned by https://github.com/zephyrproject-rtos/west (what else? (c)). It would do this sort of mapping:

zephyr -> https://github.com/zephyrproject-rtos/zephyr/  templates
# that's all for now! But enough for `west` not to be zephyr-specific.

The 2nd, actual template level should be hosted and owned by https://github.com/zephyrproject-rtos/west

template1 ->  templates/templates1
""        ->  templates/default

So in the end:

west init -t zephyr 
  => https://github.com/zephyrproject-rtos/zephyr/  templates/default

west init -t zephyr/template1
  => https://github.com/zephyrproject-rtos/zephyr/  templates/template1

Thoughts?

[marc-hb]

In a totally different issue I just mentioned https://docs.platformio.org/en/latest/frameworks/zephyr.html.

Maybe they have good ideas on this topic.

[jamie-bes]

@marc-hb The namespacing structure you described makes sense to me, though we might be starting to discuss two different (though interrelated) features here.

1. The west init -t ... command syntax, which would be a fairly simple wrapper around west init -m ... but would run an extra lookup step at the beginning to find what URL to use for the manifest repository rather than having the user pass it directly.

If the namespace for the lookup is strictly local (eg. a particular user has zephyr mapped to some official template in their local config but another user might choose to map zephyr to something totally different), then this would be a pretty small, self-contained feature, with the only additional design work being deciding how to structure the namespace configuration files (maybe using west.yml as I described earlier or maybe something simpler like an INI file). But it would require the user to maintain their own namespace configuration. However, if we wanted west init -t ... to use the same namespace for all users and work without any additional configuration from them, we'd also need the following feature:

2. A hosted registry. If this were to exist, west init -t ... could do a remote lookup to find the URL for a particular template name similar to what I alluded to with yarn create. But if we decide to do a hosted registry (or even if we do local config in the short term but plan to move to a hosted registry in the long term), controlling the namespace and preventing name conflicts becomes a bigger concern. Sorting out the design issues there and implementing the registry itself would be a challenge, which is why I was suggesting the local config option earlier, but if there's interest in a hosted registry, it would likely result in a better user experience for west init -t .... And outside of the context of project templates, a general West project registry would be extremely useful. Imagine you're working on a Zephyr application and need to use cryptography functions from Libsodium, and you could run west add zephyr/libsodium then west update to automatically add it to your west.yml and install it.

I created this issue with only feature 1 in mind, but feature 2 is pretty compelling as well and might inform our design for feature 1. But I think designing feature 2 properly would require a lot of effort and would get pretty heavy into "make West a universal package manager" territory, so if it's something that we're interested in, we should probably make a separate Github issue for it.

[marc-hb]

A hosted registry. If this were to exist, west init -t ... could do a remote lookup to find the URL for a particular template name similar to what I alluded to with yarn create.

But if you don't provide a default registry (which again, should be very easy to override on the command line), then you must provide instructions to download a "local" registry by following a... URL, so you just moved the URL problem one step further! Haven't you?

But if we decide to do a hosted registry (or even if we do local config in the short term but plan to move to a hosted registry in the long term), controlling the namespace and preventing name conflicts becomes a bigger concern.

Your scalability concerns feel even more ambitious than mine considering west has only one high-profile and well known project using it right now but... I'll roll with them! So how about org.zephyrproject -> https://github.com/zephyrproject-rtos/zephyr/ templates. Very simple and proven; namespace problem solved.

I created this issue with only feature 1 in mind, but feature 2 is pretty compelling as well and might inform our design for feature 1.

Yes, that's why I mentioned it. I agree you can start coding and testing with only a local registry as long as you keep the design open for a global one later.

[marc-hb]

@jamie-bes please take a look at

• manifest: move optional modules to a submanifest and make them optional zephyr#61505

It's not directly related but it's in the same "area".