How to add a new salt extension to the repository

[dmurphy18]

Need to document how once a new salt extension is created with copier, instruction on how to get it added to the salt-extensions repository

[lkubb]

In the past, the process regarding extensions extracted from Salt core has been:

• extract modules, including commit history, using https://github.com/salt-extensions/extension_migration
• use the Copier template to add the scaffolding
• ensure the code is updated as necessary, but don't change anything unnecessary

Then:

• ping Nicholas somewhere
• get repository + maintainer rights
• push the repository

We now have more people that can create repositories. There might also be new extensions, not based on previous modules.

Some questions we should address:

• Where should contributors request a repository?
• How much trust is required when handing them out? Afaik, organization-wide secrets can be accessed by anyone with commit access. Does trusted publishing reduce this risk (the org uses plain tokens currently)? Can we reduce it otherwise?
• Do we vet the code? If so, which criteria are we looking for? Is there a difference in the bar between extracted and newly created extensions?

Related: We will need a centralized place to put all this documentation. I'd suggest including it in the Copier template repository and hosting the built docs on GitHub Pages (salt-extensions/salt-extension-copier#17).

[dmurphy18]

@lkubb Wondering about these steps

- extract modules, including commit history, using https://github.com/salt-extensions/extension_migration
- use [the Copier template](https://github.com/salt-extensions/salt-extension-copier) to add the scaffolding

Just used the copier, did not do extension_migration, this was for a new extension for Cassandra. Wondering why need extension_migration, why not have copier do that ?, as part of it's steps.

Also the instructions in https://github.com/salt-extensions/extension_migration README

pip install copier copier-templates-extensions

don't correspond to those in copier, for example pipx install copier
The first part of extension_migration, I did automatically anyway (good hygiene habits :) )
Wondering if that README would be better to have the directory creation part and then link to copier repo readme instructions - don't have to keep things in synce and last maintenance overhead, and less chance for drift.

[lkubb]

why not have copier do that ?

We're using the Copier CLI for simplicity (writing a dedicated interface by using it as a lib would be needless work imo). Since Copier templates are mostly declarative, we would need some ugly hacks to integrate the one-off extraction step into template rendering. We could bundle the script in the salt-extension-copier repository though to avoid having separate contact points.

don't correspond to those in copier [...] Wondering if that README would be better to have the directory creation part and then link to copier repo readme instructions

Yeah, that's suboptimal. Agreed about DRY, but I think we should centralize the docs on Salt extension creation/extraction/maintenance even further, either inside salt-extension-copier or in a dedicated repository.

[max-arnold]

The main purpose of the https://github.com/salt-extensions/extension_migration script is to preserve the Git history (so git log and git blame still work), and also a bit of glob matching to help extract anything that matches e.g. *cassandra* from the main Salt repository.

Unfortunately, that tool was created before the great module purge PR and no longer works because it expects the modules to be present in the repo. One could tweak the tool and make it rewind the Salt repo checkout to a pre-purge revision, but that needs some effort and the tool still will be not very flexible.

Instead, I looked at how it works internally and described how to do the same steps manually in a much more fine-grained way. It will be included in the upcoming guide that jeanluc is going to publish soon.

[dmurphy18]

@max-arnold I thought about that, and pulled my cassandra from the 3007.x branch, given there are unlikely to be any changes to it, and if so, will fix up by hand. However, that doesn't solve the problem in general for other developers, and perhaps have to forego the github history due to the great purge (self-inflicted wound).

I was planning on looking at the changes between 3007.x and master for the files I have pulled for the extension, for a large set of files, that is quite labor intensive, but it may be a "don't care", given the extension will drift as it is developed from what was originally there.