Implement plugin-group publishing tooling with builder plugin (vmware…

…-tanzu#123)

* Implement plugin-group publishing tooling with builder plugin

Implements following commands:
 - `tanzu builder inventory plugin-group add`
 - `tanzu builder inventory plugin-group activate`
 - `tanzu builder inventory plugin-group deactivate`

The change also updates `tanzu builder plugin build` to take additional
input flag `--plugin-scope-association-file`. If this flag is provided
the build process will automatically generate a
plugin-group-manifest.yaml file within the provided `--binary-artifacts`
directory.

The change also implements `InsertPluginGroup` and
`UpdatePluginGroupActivationState` APIs in the plugininventory package

Update `cmd/plugin/builder/README.md` with the new commands.

Only install mandatory plugins using the plugin group install

cmd/plugin/builder/README.md

@@ -55,13 +55,14 @@ installation of plugins from the local directory with `tanzu plugin install --lo
 Below are the flags available with this command:
 
 ```txt
- --binary-artifacts string path to output artifacts directory (default "./artifacts")
- -h, --help help for build
- --ldflags string ldflags to set on build
- --match string match a plugin name to build, supports globbing (default "*")
- --os-arch stringArray compile for specific os-arch, use 'local' for host os, use '<os>_<arch>' for specific (default [all])
- --path string path of plugin directory (default "./cmd/plugin")
- -v, --version string version of the plugins
+ --binary-artifacts string path to output artifacts directory (default "./artifacts")
+ -h, --help help for build
+ --ldflags string ldflags to set on build
+ --match string match a plugin name to build, supports globbing (default "*")
+ --os-arch stringArray compile for specific os-arch, use 'local' for host os, use '<os>_<arch>' for specific (default [all])
+ --path string path of plugin directory (default "./cmd/plugin")
+ --plugin-scope-association-file string file specifying plugin scope association
+ -v, --version string version of the plugins
 ```
 
 Below are the examples:
@@ -77,7 +78,44 @@ Below are the examples:
  tanzu builder plugin build --path ./cmd/plugin --version v0.0.2 --os-arch all --match foo
 ```
 
-Here is the artifacts directory structure that gets generated by running the `tanzu builder plugin build --path ./cmd/plugin --version v0.0.2 --os-arch darwin_amd64 --os-arch linux_amd64` command. Assuming we have 2 plugins `foo` with a `global` target and `bar` with a `kubernetes` target available under `./cmd/plugin` directory.
+The `tanzu builder plugin build` command provides a convenient way to create a [plugin-group manifest file](#inventory-plugin-group-add) (`plugin_group_manifest.yaml`) containing plugin-group metadata by providing the `--plugin-scope-association-file` flag. The purpose of a plugin-group is to define a product-release-specific set of plugins for users to easily install plugins for the specific product release. More details are provided in the [inventory-plugin-group-add](#inventory-plugin-group-add) section.
+
+Using the `--plugin-scope-association-file` flag is a convenient way to generate a plugin-group manifest file consisting of the plugins built in the `artifacts` directory. However, if any external plugins or different versions of plugins need to be included in the plugin-group manifest file, the developer will need to manually create this file. When the `--plugin-scope-association-file` flag is provided, the tooling will generate the `plugin_group_manifest.yaml` file within the same binary artifacts directory.
+
+Below is a sample `plugin-scope-association.yaml`:
+
+```yaml
+plugins:
+- name: foo
+ target: global
+ isContextScoped: false
+- name: bar
+ target: kubernetes
+ isContextScoped: true
+- name: baz
+ target: mission-control
+ isContextScoped: true
+```
+
+Below is the `plugin_group_manifest.yaml` that will get generated based on the above `plugin-scope-association.yaml`:
+
+```yaml
+plugins:
+- name: foo
+ target: global
+ isContextScoped: false
+ version: v0.0.2
+- name: bar
+ target: kubernetes
+ isContextScoped: true
+ version: v0.0.2
+- name: baz
+ target: mission-control
+ isContextScoped: true
+ version: v0.0.2
+```
+
+Here is the artifacts directory structure that gets generated by running the `tanzu builder plugin build --path ./cmd/plugin --version v0.0.2 --os-arch darwin_amd64 --os-arch linux_amd64 --plugin-scope-association-file ./cmd/plugin/plugin-scope-association.yaml` command. Assuming we have 2 plugins `foo` with a `global` target and `bar` with a `kubernetes` target available under `./cmd/plugin` directory.
 
 ```shell
 artifacts
@@ -103,7 +141,8 @@ artifacts
 │  │    └── v0.0.2
 │  │      └── tanzu-bar-linux_amd64
 │  └── plugin_manifest.yaml
-└── plugin_manifest.yaml
+├── plugin_manifest.yaml
+└── plugin_group_manifest.yaml
 
 ```
 
@@ -235,3 +274,55 @@ Below are the examples:
  # Deactivate plugins in the inventory database based on the specified manifest file
  tanzu builder inventory plugin deactivate --repository localhost:5002/test/v1/tanzu-cli/plugins --vendor vmware --publisher tkg1 --manifest ./artifacts/packages/plugin_manifest.yaml
 ```
+
+### Inventory-plugin-group-add
+
+Once the plugins are published and added to the inventory database the next thing would be to add/create plugin-groups. The purpose of a plugin-group is to define a product-release-specific set of plugins for users to easily install plugins for the specific product release. To support this use-case the `builder` plugin provides a `tanzu builder inventory plugin-group add` command.
+
+Below are the flags available with the `tanzu builder inventory plugin-group add` command:
+
+```txt
+ --deactivate mark plugin-group as deactivated
+ -h, --help help for add
+ --manifest string manifest file specifying plugin-group details that needs to be processed
+ --name string name of the plugin-group
+ --override override the plugin-group if already exists
+ --plugin-inventory-image-tag string tag to which plugin inventory image needs to be published (default "latest")
+ --publisher string name of the publisher
+ --repository string repository to publish plugin inventory image
+ --vendor string name of the vendor
+```
+
+Below are some examples:
+
+```shell
+ # Add plugin-group entries to the inventory database based on the specified plugin-group manifest file
+ tanzu builder inventory plugin-group add --name v1.0.0 --repository project-stg.registry.vmware.com/test/v1/tanzu-cli/plugins --vendor vmware --publisher tkg --manifest ./artifacts/plugins/plugin_group_manifest.yaml
+```
+
+Here the `--manifest` flag is used to provide metadata about the plugin-group including which plugins to associate with the plugin-group.
+
+### Inventory-plugin-group-activate-deactivate
+
+In some scenarios, such as preparing for a new product release, a plugin-group may need to be created and added to the inventory database but kept "deactivated". A "deactivated" plugin-group is not visible to the Tanzu CLI and therefore will not be discovered by users before the official product release, however testers can configure the CLI to discover "deactivated" plugins. To support this scenario the `builder` plugin implements the `tanzu builder inventory plugin-group activate` and `tanzu builder inventory plugin-group deactivate` commands.
+
+Below are the flags available with `tanzu builder inventory plugin-group activate` and `tanzu builder inventory plugin-group deactivate` commands:
+
+```txt
+ -h, --help help for activate
+ --name string name of the plugin group
+ --plugin-inventory-image-tag string tag to which plugin inventory image needs to be published (default "latest")
+ --publisher string name of the publisher
+ --repository string repository to publish plugin inventory image
+ --vendor string name of the vendor
+```
+
+Below are some examples:
+
+```shell
+ # Activate plugin-group in the inventory database
+ tanzu builder inventory plugin-group activate --name v1.0.0 --repository localhost:5002/test/v1/tanzu-cli/plugins --vendor vmware --publisher tkg1
+
+ # Dectivate plugin-group in the inventory database
+ tanzu builder inventory plugin-group deactivate --name v1.0.0 --repository localhost:5002/test/v1/tanzu-cli/plugins --vendor vmware --publisher tkg1
+```

cmd/plugin/builder/command/cli_compile.go

@@ -54,16 +54,17 @@ type plugin struct {
 
 // PluginCompileArgs contains the values to use for compiling plugins.
 type PluginCompileArgs struct {
- Version string
- SourcePath string
- ArtifactsDir string
- LDFlags string
- Tags string
- Match string
- Description string
- GoPrivate string
- TargetArch []string
- GroupByOSArch bool
+ Version string
+ SourcePath string
+ ArtifactsDir string
+ LDFlags string
+ Tags string
+ Match string
+ Description string
+ GoPrivate string
+ PluginScopeAssociationFile string
+ TargetArch []string
+ GroupByOSArch bool
 }
 
 const local = "local"
@@ -220,6 +221,11 @@ func Compile(compileArgs *PluginCompileArgs) error {
  return err
  }
 
+ err = savePluginGroupManifest(manifest.Plugins, compileArgs.ArtifactsDir, compileArgs.PluginScopeAssociationFile, compileArgs.GroupByOSArch)
+ if err != nil {
+ return err
+ }
+
  log.Success("successfully built local repository")
  return nil
 }
@@ -596,3 +602,59 @@ func savePluginManifest(manifest cli.Manifest, artifactsDir string, groupByOSArc
  }
  return nil
 }
+
+func savePluginGroupManifest(plugins []cli.Plugin, artifactsDir, pluginScopeAssociationFile string, groupByOSArch bool) error {
+ if !groupByOSArch || pluginScopeAssociationFile == "" {
+ return nil
+ }
+
+ log.Info("saving plugin group manifest...")
+
+ b, err := os.ReadFile(pluginScopeAssociationFile)
+ if err != nil {
+ return err
+ }
+
+ psm := &cli.PluginScopeMetadata{}
+ err = yaml.Unmarshal(b, psm)
+ if err != nil {
+ return err
+ }
+
+ getPluginScope := func(name, target string) bool {
+ for _, p := range psm.Plugins {
+ if p.Name == name && p.Target == target {
+ return p.IsContextScoped
+ }
+ }
+ return false
+ }
+
+ pgManifest := cli.PluginGroupManifest{
+ CreatedTime: time.Now(),
+ Plugins: []cli.PluginNameTargetScopeVersion{},
+ }
+
+ for _, plug := range plugins {
+ pluginNTSV := cli.PluginNameTargetScopeVersion{}
+ pluginNTSV.Name = plug.Name
+ pluginNTSV.Target = plug.Target
+ pluginNTSV.IsContextScoped = getPluginScope(plug.Name, plug.Target)
+ pluginNTSV.Version = plug.Versions[0]
+
+ pgManifest.Plugins = append(pgManifest.Plugins, pluginNTSV)
+ }
+
+ b, err = yaml.Marshal(pgManifest)
+ if err != nil {
+ return err
+ }
+
+ manifestPath := filepath.Join(artifactsDir, cli.PluginGroupManifestFileName)
+ err = os.WriteFile(manifestPath, b, 0644)
+ if err != nil {
+ return err
+ }
+
+ return nil
+}

cmd/plugin/builder/helpers/helper.go

@@ -32,6 +32,21 @@ func ReadPluginManifest(pluginManifestFile string) (*cli.Manifest, error) {
  return pluginManifest, nil
 }
 
+// ReadPluginGroupManifest reads the PluginGroupManifest file and returns PluginGroupManifest object
+func ReadPluginGroupManifest(pluginGroupManifestFile string) (*cli.PluginGroupManifest, error) {
+ data, err := os.ReadFile(pluginGroupManifestFile)
+ if err != nil {
+ return nil, errors.Wrap(err, "fail to read the plugin-group manifest file")
+ }
+
+ pluginGroupManifest := &cli.PluginGroupManifest{}
+ err = yaml.Unmarshal(data, pluginGroupManifest)
+ if err != nil {
+ return nil, errors.Wrap(err, "fail to read the plugin-group manifest file")
+ }
+ return pluginGroupManifest, nil
+}
+
 // GetPluginArchiveRelativePath creates plugin archive relative path from metadata
 func GetPluginArchiveRelativePath(plugin cli.Plugin, osArch cli.Arch, version string) string {
  pluginTarFileName := fmt.Sprintf("%s-%s.tar.gz", plugin.Name, osArch.String())

cmd/plugin/builder/inventory.go

@@ -23,6 +23,7 @@ func newInventoryCmd() *cobra.Command {
  inventoryCmd.AddCommand(
  newInventoryInitCmd(),
  newInventoryPluginCmd(),
+ newInventoryPluginGroupCmd(),
  )
 
  return inventoryCmd

cmd/plugin/builder/inventory/inventory_helper.go

@@ -0,0 +1,29 @@
+// Copyright 2023 VMware, Inc. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+package inventory
+
+import (
+ "path/filepath"
+
+ "github.com/pkg/errors"
+
+ "github.com/vmware-tanzu/tanzu-cli/cmd/plugin/builder/imgpkg"
+ "github.com/vmware-tanzu/tanzu-cli/pkg/plugininventory"
+)
+
+func inventoryDBDownload(imgpkgOptions imgpkg.ImgpkgWrapper, pluginInventoryDBImage, tempDir string) (string, error) {
+ err := imgpkgOptions.PullImage(pluginInventoryDBImage, tempDir)
+ if err != nil {
+ return "", errors.Wrapf(err, "error while pulling database from the image: %q", pluginInventoryDBImage)
+ }
+ return filepath.Join(tempDir, plugininventory.SQliteDBFileName), nil
+}
+
+func inventoryDBUpload(imgpkgOptions imgpkg.ImgpkgWrapper, pluginInventoryDBImage, dbFile string) error {
+ err := imgpkgOptions.PushImage(pluginInventoryDBImage, dbFile)
+ if err != nil {
+ return errors.Wrapf(err, "error while publishing inventory database to the repository as image: %q", pluginInventoryDBImage)
+ }
+ return nil
+}