Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix the generated DABs JSON schema #1322

Merged
merged 10 commits into from
Mar 28, 2024
Merged

Fix the generated DABs JSON schema #1322

merged 10 commits into from
Mar 28, 2024

Conversation

shreyas-goenka
Copy link
Contributor

@shreyas-goenka shreyas-goenka commented Mar 27, 2024 •
edited
Loading

Changes

This PR fixes bundle schema being broken because for_each_task: null was set in the generated schema. This is not valid according to the JSON schema specification and thus the Red Hat YAML VSCode extension was failing to parse the YAML configuration.

This PR fixes: #1312

Tests

The fix itself was tested manually. I asserted that the autocompletion works now. This was mistakenly overlooked the first time around when the regression was introduced in #1204 because the YAML extension provides best-effort autocomplete suggestions even if the JSON schema fails to load.

To prevent future regressions we also add a test to assert that the JSON schema generated itself is a valid JSON schema object. This is done via using the ajv-cli to validate the schema. This package is also used by the Red Hat YAML extension and thus provides a high fidelity check for ensuring the JSON schema is valid.

Before, with the old schema:

shreyas.goenka@THW32HFW6T cli-versions % ajv validate -s proj/schema-216.json -d ../bundle-playground-3/databricks.yml
schema proj/schema-216.json is invalid
error: schema is invalid: data/properties/resources/properties/jobs/additionalProperties/properties/tasks/items/properties/for_each_task must be object,boolean, data/properties/resources/properties/jobs/additionalProperties/properties/tasks/items must be array, data/properties/resources/properties/jobs/additionalProperties/properties/tasks/items must match a schema in anyOf

After, with the new schema:

shreyas.goenka@THW32HFW6T cli-versions % ajv validate -s proj/schema-dev.json -d ../bundle-playground-3/databricks.yml
../bundle-playground-3/databricks.yml valid

After, autocomplete suggestions:

Screenshot 2024-03-27 at 6 35 57 PM

pietern
pietern approved these changes Mar 28, 2024
View reviewed changes
.github/workflows/push.yml Outdated Show resolved Hide resolved
@pietern
Copy link
Contributor

pietern commented Mar 28, 2024

Thanks for fixing and adding the integration test. This was a blind spot for sure.

@shreyas-goenka shreyas-goenka added this pull request to the merge queue Mar 28, 2024
Merged via the queue into main with commit cddc5f9 Mar 28, 2024
5 checks passed
@shreyas-goenka shreyas-goenka deleted the fix/bundle-schema branch March 28, 2024 11:34
andrewnester added a commit that referenced this pull request Apr 3, 2024
@andrewnester
Release v0.217.0
1f85841 
CLI:
 * Added `auth describe` command ([#1244](#1244)).
 * Fixed message for successful auth describe run ([#1336](#1336)).

Bundles:
 * Make bundle validation print text output by default ([#1335](#1335)).
 * Use UserName field to identify if service principal is used ([#1310](#1310)).
 * Allow unknown properties in the config file for template initialization ([#1315](#1315)).
 * Remove support for DATABRICKS_BUNDLE_INCLUDES ([#1317](#1317)).
 * Make `bundle.deployment` optional in the bundle schema ([#1321](#1321)).
 * Add allow list for resources when bundle `run_as` is set ([#1233](#1233)).
 * Fix the generated DABs JSON schema ([#1322](#1322)).
 * Make bundle loaders return diagnostics ([#1319](#1319)).
 * Add `bundle debug terraform` command ([#1294](#1294)).
 * Allow specifying CLI version constraints required to run the bundle ([#1320](#1320)).

Internal:
 * Retain location information of variable reference ([#1333](#1333)).
 * Define `dyn.Mapping` to represent maps ([#1301](#1301)).
 * Return `diag.Diagnostics` from mutators ([#1305](#1305)).
 * Fix flaky test in `libs/process` ([#1314](#1314)).
 * Move path field to bundle type ([#1316](#1316)).
 * Load bundle configuration from mutator ([#1318](#1318)).
 * Return diagnostics from `config.Load` ([#1324](#1324)).
 * Return warning for nil primitive types during normalization ([#1329](#1329)).
 * Include `dyn.Path` in normalization warnings and errors ([#1332](#1332)).
 * Make normalization return warnings instead of errors ([#1334](#1334)).

API Changes:
 * Added `databricks lakeview migrate` command.
 * Added `databricks lakeview unpublish` command.
 * Changed `databricks ip-access-lists get` command . New request type is .

OpenAPI commit e316cc3d78d087522a74650e26586088da9ac8cb (2024-04-03)
Dependency updates:
 * Bump github.com/databricks/databricks-sdk-go from 0.36.0 to 0.37.0 ([#1326](#1326)).
@andrewnester andrewnester mentioned this pull request Apr 3, 2024
Merged
github-merge-queue bot pushed a commit that referenced this pull request Apr 3, 2024
@andrewnester
Release v0.217.0 (#1337)
6ac45e8 
Breaking Change:
* Add allow list for resources when bundle `run_as` is set
([#1233](#1233)).
* Make bundle validation print text output by default
([#1335](#1335)).

CLI:
* Added `auth describe` command
([#1244](#1244)).
* Fixed message for successful auth describe run
([#1336](#1336)).

Bundles:
* Use UserName field to identify if service principal is used
([#1310](#1310)).
* Allow unknown properties in the config file for template
initialization ([#1315](#1315)).
* Remove support for DATABRICKS_BUNDLE_INCLUDES
([#1317](#1317)).
* Make `bundle.deployment` optional in the bundle schema
([#1321](#1321)).
* Fix the generated DABs JSON schema
([#1322](#1322)).
* Make bundle loaders return diagnostics
([#1319](#1319)).
* Add `bundle debug terraform` command
([#1294](#1294)).
* Allow specifying CLI version constraints required to run the bundle
([#1320](#1320)).

Internal:
* Retain location information of variable reference
([#1333](#1333)).
* Define `dyn.Mapping` to represent maps
([#1301](#1301)).
* Return `diag.Diagnostics` from mutators
([#1305](#1305)).
* Fix flaky test in `libs/process`
([#1314](#1314)).
* Move path field to bundle type
([#1316](#1316)).
* Load bundle configuration from mutator
([#1318](#1318)).
* Return diagnostics from `config.Load`
([#1324](#1324)).
* Return warning for nil primitive types during normalization
([#1329](#1329)).
* Include `dyn.Path` in normalization warnings and errors
([#1332](#1332)).
* Make normalization return warnings instead of errors
([#1334](#1334)).
API Changes:
 * Added `databricks lakeview migrate` command.
 * Added `databricks lakeview unpublish` command.
* Changed `databricks ip-access-lists get` command . New request type is
.

OpenAPI commit e316cc3d78d087522a74650e26586088da9ac8cb (2024-04-03)
Dependency updates:
* Bump github.com/databricks/databricks-sdk-go from 0.36.0 to 0.37.0
([#1326](#1326)).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andrewnester andrewnester andrewnester approved these changes

@pietern pietern pietern approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

databricks bundle schema does no longer work with visual studio code redhat-yaml extension
3 participants
@shreyas-goenka @pietern @andrewnester