From cddc5f97f862805d885ea3b0d1d497d51e0a59fa Mon Sep 17 00:00:00 2001 From: shreyas-goenka <88374338+shreyas-goenka@users.noreply.github.com> Date: Thu, 28 Mar 2024 16:55:36 +0530 Subject: [PATCH] Fix the generated DABs JSON schema (#1322) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Changes This PR fixes bundle schema being broken because `for_each_task: null` was set in the generated schema. This is not valid according to the JSON schema specification and thus the Red Hat YAML VSCode extension was failing to parse the YAML configuration. This PR fixes: https://github.com/databricks/cli/issues/1312 ## Tests The fix itself was tested manually. I asserted that the autocompletion works now. This was mistakenly overlooked the first time around when the regression was introduced in https://github.com/databricks/cli/pull/1204 because the YAML extension provides best-effort autocomplete suggestions even if the JSON schema fails to load. To prevent future regressions we also add a test to assert that the JSON schema generated itself is a valid JSON schema object. This is done via using the `ajv-cli` to validate the schema. This package is also used by the Red Hat YAML extension and thus provides a high fidelity check for ensuring the JSON schema is valid. Before, with the old schema: ``` shreyas.goenka@THW32HFW6T cli-versions % ajv validate -s proj/schema-216.json -d ../bundle-playground-3/databricks.yml schema proj/schema-216.json is invalid error: schema is invalid: data/properties/resources/properties/jobs/additionalProperties/properties/tasks/items/properties/for_each_task must be object,boolean, data/properties/resources/properties/jobs/additionalProperties/properties/tasks/items must be array, data/properties/resources/properties/jobs/additionalProperties/properties/tasks/items must match a schema in anyOf ``` After, with the new schema: ``` shreyas.goenka@THW32HFW6T cli-versions % ajv validate -s proj/schema-dev.json -d ../bundle-playground-3/databricks.yml ../bundle-playground-3/databricks.yml valid ``` After, autocomplete suggestions: Screenshot 2024-03-27 at 6 35 57 PM --- .github/workflows/push.yml | 26 ++++++++++++++++++++++++++ bundle/schema/schema.go | 4 +++- 2 files changed, 29 insertions(+), 1 deletion(-) diff --git a/.github/workflows/push.yml b/.github/workflows/push.yml index 18ba54a379..244bdeee5b 100644 --- a/.github/workflows/push.yml +++ b/.github/workflows/push.yml @@ -89,3 +89,29 @@ jobs: run: | # Exit with status code 1 if there are differences (i.e. unformatted files) git diff --exit-code + + validate-bundle-schema: + runs-on: ubuntu-latest + + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Setup Go + uses: actions/setup-go@v5 + with: + go-version: 1.21.x + + # Github repo: https://github.com/ajv-validator/ajv-cli + - name: Install ajv-cli + run: npm install -g ajv-cli@5.0.0 + + # Assert that the generated bundle schema is a valid JSON schema by using + # ajv-cli to validate it against a sample configuration file. + # By default the ajv-cli runs in strict mode which will fail if the schema + # itself is not valid. Strict mode is more strict than the JSON schema + # specification. See for details: https://ajv.js.org/options.html#strict-mode-options + - name: Validate bundle schema + run: | + go run main.go bundle schema > schema.json + ajv -s schema.json -d ./bundle/tests/basic/databricks.yml diff --git a/bundle/schema/schema.go b/bundle/schema/schema.go index 7153f38f60..b37f72d9b6 100644 --- a/bundle/schema/schema.go +++ b/bundle/schema/schema.go @@ -95,7 +95,9 @@ func safeToSchema(golangType reflect.Type, docs *Docs, traceId string, tracker * // HACK to unblock CLI release (13th Feb 2024). This is temporary until proper // support for recursive types is added to the schema generator. PR: https://github.com/databricks/cli/pull/1204 if traceId == "for_each_task" { - return nil, nil + return &jsonschema.Schema{ + Type: jsonschema.ObjectType, + }, nil } // WE ERROR OUT IF THERE ARE CYCLES IN THE JSON SCHEMA