tools/importer-rest-api-specs: refactoring the Parser package (#4307)

* `tools/importer-rest-api-specs`: refactoring the `commonschema` Matchers out into the new structure

* `tools/importer-rest-api-specs`: refactoring the Constant package

* `tools/importer-rest-api-specs`: porting the Network workaround into a Data Workaround

* `tools/importer-rest-api-specs`: porting the `dataworkarounds` package over to using the SDK types/new path

This is a copy and patch up, since I want to leave the existing code path as-is for the moment, to break it apart more effectively

* `tools/importer-rest-api-specs`: scaffolding out the logic to parse a Service/APIVersion

* `tools/importer-rest-api-specs`: porting over the "remove unused items" logic

* `tools/importer-rest-api-specs`: adding a shim to the updated `dataworkarounds` package

* dependencies: updating to `v0.69.0` of `github.com/hashicorp/go-azure-helpers`

* `tools/importer-rest-api-specs`: refactoring the `normalization` package into the new structure

* `tools/importer-rest-api-specs`: moving all of the `cleanup` items into the new package

This wants refactoring, but moving it over allows cleaning the rest up

* `tools/importer-rest-api-specs`: adding inner most sdk operation object definition

* `tools/importer-rest-api-specs`: moving the `commonids` into a subpackage

There's a bunch of these, and whilst these'll get replaced in time with #4017
but for now this cleans this up

* `tools/data-api-sdk`: removing an empty todo file

* `tools/importer-rest-api-specs`: refactoring more logic out to the new package

The `parser` package itself needs cleaning, but there's a few things to do first

* `tools/importer-rest-api-specs`: moving another `ignore` function

* WIP

* tools/importer-rest-api-specs: reimplementing #4290 atop the rebase

The rebase is messed up, but it's a minor patch so should be fine

* `tools/importer-rest-api-specs`: adding a placeholder for parsing an APIResource

* `tools/importer-rest-api-specs`: refactoring the parser into smaller packages

This leaves the old parser logic in place for now, which'll be cleaned up in the next commit

* `tools/importer-rest-api-specs`: fixing an issue where the `validate` command didn't pass the API Definitions Directory correctly

* `tools/importer-rest-api-specs`: title-casing the ReferenceName - which ensures that when we compare the ObjectDefinition from a previously parsed/currently parsed dataset we have matching values

These are normalised post-parsing, so when comparing against the old data set we need to use TitleCase too, so we might as well do this here

* `tools/importer-rest-api-specs`: ensuring that TypeSpec examples are filtered out

* `tools/importer-rest-api-specs`: trimming `*.json` from the filename to build the api resource name

* `tools/importer-rest-api-specs`: removing (most) of the old code path

* `tools/data-api-sdk`: making IsCommonType required so this is more explicit

* `tools/importer-rest-api-specs`: fixing the validation

* `tools/importer-rest-api-specs`: re-enabling most of the tests

* `tools/importer-rest-api-specs`: consolidating/porting over the data tests

These ensure that the Configs can be parsed and that the Data from them can be loaded

* `tools/importer-rest-api-specs`: fixing the test format

* tools/importer-rest-api-specs: skipping the local test

* `tools/importer-rest-api-specs`: fixing the non-discriminator tests

* `tools/importer-rest-api-specs`: updating/reworking the supplementary data test

@manicminer is going to work through fixing this one up, this requires parsing the data identifying any parent types and patching any disciminated implementations across

* `importer-rest-api-specs`: fix up discriminator handling for datafactory

* `importer-rest-api-specs`: shelve SupplementaryData support for now, since it doesn't seem to be required for DataFactory models as they use external Swagger Refs.

* `importer-rest-api-specs`: implement `-services` argument

* `importer-rest-api-specs`: operate on maps instead of passing them around

* `importer-rest-api-specs`: normalize swagger tag names to prevent duplicates (avoid parsing the same tag twice)

* `importer-rest-api-specs`: normalize API resources after parsing all of them

* `importer-rest-api-specs`: ensure new APIResource is populated

* `importer-rest-api-specs`: maps are pointers

* `importer-rest-api-specs`: dereference more maps

* `importer-rest-api-specs`: remove use of strings.Title, dereference more maps

* limit scope of SDK generation test to the SDK being tested

* `importer-rest-api-specs`: two loops become one

* `importer-rest-api-specs`: workaround inconsistent casing of "VirtualWANs"

* Revert "`importer-rest-api-specs`: two loops become one"

This reverts commit 31b45ca.

* `importer-rest-api-specs`: normalize "attachednetworks" segment of resource IDs

* Revert "`tools/data-api-sdk`: making IsCommonType required so this is more explicit"

This reverts commit 71a166f.

* importer-rest-api-specs: use updated forks for `go-openapi/analysis`, `go-openapi/jsonreference`, and `go-openapi/spec`

* importer-rest-api-specs: remove Supplementary Data parser as is not used/needed

* importer-rest-api-specs: rework the swagger spec parser and manually resolve all remote references

* importer-rest-api-specs: fix format string, ensure model names are URL-decoded when parsing from the Ref URL fragment

* importer-rest-api-specs: updated tests for discriminated models to expect full ancestry

* importer-rest-api-specs: remove vestiges of supplementary data spike

* importer-rest-api-specs: do not set the description, see #4434

* importer-rest-api-specs: hack for conflicting constant names in the same set of resources

* importer-rest-api-specs: disable check for non-default Resource Provider when parsing operations, as this was never utilised and leads to skipping resources/operations

* upper case URL in the go sdk

---------

Co-authored-by: Tom Bamford <[email protected]>
Co-authored-by: Steph <[email protected]>

scripts/automation-generate-go-sdk.sh

@@ -6,17 +6,21 @@
 set -e
 
 DIR="$(cd "$(dirname "$0")" && pwd)/.."
-
 sdkToGenerate="${1}"
-if [[ "${sdkToGenerate}" == "" ]]; then
-  echo "must specify SDK to generate!" >&2
+
+usage() {
+  echo "Usage: automation-generate-go-sdk.sh sdkToGenerate" >&2
   echo "" >&2
-  echo "supported values are:" >&2
+  echo "sdkToGenerate is required and should be one of:" >&2
   echo "  microsoft-graph" >&2
   echo "  resource-manager" >&2
   echo "" >&2
-  echo "example usage:" >&2
-  echo "${0} resource-manager" >&2
+}
+
+if [[ "${sdkToGenerate}" == "" ]]; then
+  echo "sdkToGenerate not specified" >&2
+  echo "" >&2
+  usage
   exit 1
 fi
 

tools/data-api-sdk/v1/helpers/inner_most_operation_object_definition.go

@@ -0,0 +1,21 @@
+// Copyright (c) HashiCorp, Inc.
+// SPDX-License-Identifier: MPL-2.0
+
+package helpers
+
+import "github.com/hashicorp/pandora/tools/data-api-sdk/v1/models"
+
+// InnerMostSDKOperationOptionObjectDefinition returns the inner most SDKOperationOptionObjectDefinition.
+//
+// In the event of an SDKOperationOptionObjectDefinition with no NestedItem, the current item will be returned.
+// In the event of a NestedItem being present, or a NestedItem having a NestedItem, this method
+// will recurse until it finds the SDKOperationOptionObjectDefinition without a NestedItem.
+//
+// This is useful for obtaining the inner type for assignment purposes.
+func InnerMostSDKOperationOptionObjectDefinition(input models.SDKOperationOptionObjectDefinition) models.SDKOperationOptionObjectDefinition {
+	if input.NestedItem != nil {
+		return InnerMostSDKOperationOptionObjectDefinition(*input.NestedItem)
+	}
+
+	return input
+}

tools/importer-rest-api-specs/GNUmakefile

@@ -16,6 +16,9 @@ import: build
 import-with-api: build
 	./importer-rest-api-specs import --data-api=http://localhost:8080
 
+validate: build
+	./importer-rest-api-specs validate
+
 test: build
 	go test -v ./... -timeout=60m