Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adapt the SOR for managed upgrades #150312

Closed
3 tasks done
pgayvallet opened this issue Feb 6, 2023 · 1 comment
Closed
3 tasks done

Adapt the SOR for managed upgrades #150312

pgayvallet opened this issue Feb 6, 2023 · 1 comment
Labels
Epic:ZDTmigrations Zero downtime migrations Feature:Migrations Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc

Comments

@pgayvallet
Copy link
Contributor

pgayvallet commented Feb 6, 2023 •
edited by TinaHeiligers
Loading

Part of #150296

This issue is here to track the changes required by managed upgrades in the SOR implementation.

(Each bullet point should be an issue, they will be created when necessary)

MVP

Short term

  • Allowing to specify the version to return from the SOR APIs (add as a parameter)

Later / eventually
This was referenced Feb 6, 2023
Closed
Closed
@botelastic botelastic bot added the needs-team Issues missing a team label label Feb 6, 2023
@pgayvallet pgayvallet added Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc Epic:ZDTmigrations Zero downtime migrations labels Feb 6, 2023
@botelastic botelastic bot removed the needs-team Issues missing a team label label Feb 6, 2023
@pgayvallet pgayvallet added needs-team Issues missing a team label Feature:Migrations labels Feb 6, 2023
@botelastic botelastic bot removed the needs-team Issues missing a team label label Feb 6, 2023
This was referenced Mar 1, 2023
Merged
Closed
Closed
Closed
This was referenced Mar 27, 2023
Closed
Closed
@pgayvallet pgayvallet mentioned this issue Mar 30, 2023
Merged
pgayvallet added a commit that referenced this issue Apr 3, 2023
@pgayvallet @kibanamachine
DocumentMigration: implement downward transformation of documents (#1…
54df909 
…53921)

## Summary

Fix #150301

Implements `DocumentMigrator.transformDown` and the associated
transformation pipeline.

Note: The feature is currently unused, and will only be when we'll start
#150312

---------

Co-authored-by: kibanamachine <[email protected]>
@pgayvallet pgayvallet mentioned this issue May 16, 2023
Merged
pgayvallet added a commit that referenced this issue May 23, 2023
@pgayvallet @kibanamachine
[ZDT] DocumentMigrator: support higher version documents (#157895)
c24dc35 
## Summary

Part of #150312
(next steps depend on #153117)

**This PR does two things:**
- introduce the concept of version persistence schema
- adapt the document migrator to support downward migrations for
documents of an higher version.

In the follow-up, we will then update the calls from the SOR to the
document migrator to allow downward conversions when we're using the ZDT
migration algorithm (which requires
#153117 to be merged)

### Model version persistence schema.

*(This is what has also been named 'eviction schema' or 'known fields
schema'.)*

A new `SavedObjectsModelVersion.schemas.backwardConversion` property was
added to the model version definition.

This 'schema' can either be an arbitrary function, or a `schema.object`
from `@kbn/config-schema`

```ts
type SavedObjectModelVersionBackwardConversionSchema<
  InAttrs = unknown,
  OutAttrs = unknown
> = ObjectType | SavedObjectModelVersionBackwardConversionFn<InAttrs, OutAttrs>;
```

When specified for a version, the document's attributes will go thought
this schema during down conversions by the document migrator.

### Adapt the document migrator to support downward migrations for
documents of an higher version.

Add an `allowDowngrade` option to `DocumentMigrator.migrate` and
`KibanaMigrator.migrateDocument`. When this option is set to `true`, the
document migration will accept to 'downgrade' the document if necessary,
instead of throwing an error as done when the option is `false` or
unspecified (which was the only behavior prior to this PR's changes)

---------

Co-authored-by: kibanamachine <[email protected]>
@pgayvallet pgayvallet mentioned this issue May 23, 2023
Merged
delanni pushed a commit to delanni/kibana that referenced this issue May 25, 2023
@pgayvallet @kibanamachine @delanni
[ZDT] DocumentMigrator: support higher version documents (elastic#157895
75d8f47 
)

## Summary

Part of elastic#150312
(next steps depend on elastic#153117)

**This PR does two things:**
- introduce the concept of version persistence schema
- adapt the document migrator to support downward migrations for
documents of an higher version.

In the follow-up, we will then update the calls from the SOR to the
document migrator to allow downward conversions when we're using the ZDT
migration algorithm (which requires
elastic#153117 to be merged)

### Model version persistence schema.

*(This is what has also been named 'eviction schema' or 'known fields
schema'.)*

A new `SavedObjectsModelVersion.schemas.backwardConversion` property was
added to the model version definition.

This 'schema' can either be an arbitrary function, or a `schema.object`
from `@kbn/config-schema`

```ts
type SavedObjectModelVersionBackwardConversionSchema<
  InAttrs = unknown,
  OutAttrs = unknown
> = ObjectType | SavedObjectModelVersionBackwardConversionFn<InAttrs, OutAttrs>;
```

When specified for a version, the document's attributes will go thought
this schema during down conversions by the document migrator.

### Adapt the document migrator to support downward migrations for
documents of an higher version.

Add an `allowDowngrade` option to `DocumentMigrator.migrate` and
`KibanaMigrator.migrateDocument`. When this option is set to `true`, the
document migration will accept to 'downgrade' the document if necessary,
instead of throwing an error as done when the option is `false` or
unspecified (which was the only behavior prior to this PR's changes)

---------

Co-authored-by: kibanamachine <[email protected]>
pgayvallet added a commit that referenced this issue May 30, 2023
@pgayvallet
[ZDT] SOR: handle higher-version documents retrieved from persistence. (
50950e9 
#158251)

## Summary

Follow-up of #157895
Part of #150312

Adapt the SOR to accept retrieving documents on higher versions from the
persistence, and convert them back to the latest knows version (using
the version schema feature added in
#157895).
@rudolf
Copy link
Contributor

rudolf commented Oct 12, 2023

Closing as we have #165434 for tracking the last remaining piece of this work.

@rudolf rudolf closed this as completed Oct 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Epic:ZDTmigrations Zero downtime migrations Feature:Migrations Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rudolf @pgayvallet