HDDS-11465. Introducing Schema Versioning for Recon to Handle Fresh Installs and Upgrades.

[ArafatKhan2198]

What changes were proposed in this pull request?

This PR introduces an upgrade framework in Recon that mirrors the pattern used in OMLayoutFeature with some key differences tailored for Recon's needs. The primary purpose of this PR is to manage and handle schema upgrades automatically during Recon startup, storing layout version information on disk and executing necessary upgrade actions.

Summary of Changes:

• Replication of OMLayoutFeature Pattern: Recon now follows a similar schema upgrade pattern as used in the OM layer, but with auto-finalization. Unlike in OM, Recon will automatically finalize on restart without requiring manual intervention.
• Storage of Metadata Value: The Metadata Layout Version (MLV) is stored in the RECON_SCHEMA_VERSION table on disk. This table tracks the schema version to identify which features have been applied.
• Computation of SLV: The Software Layout Version (SLV) is computed dynamically based on the latest version defined in ReconLayoutFeature at runtime.
• Code Flow During Startup :-
  • On Recon startup, the ReconLayoutVersionManager fetches the current MLV from the RECON_SCHEMA_VERSION table.
  • It then computes the SLV by identifying the highest version in ReconLayoutFeature.
  • The manager finalizes any layout features that require upgrading by executing their associated actions and updates the MLV.
• Newly Added Classes :-
  • ReconLayoutVersionManager: Handles the determination of the current MLV and SLV and performs the finalization of pending layout features.
  • ReconSchemaVersionTableManager: Manages interactions with the RECON_SCHEMA_VERSION table, including fetching and updating the current MLV.
  • ReconLayoutFeature: Enum defining the layout features with their version, description, and direct integration of upgrade actions.
• Differences from OMLayoutFeature:
  • No manual finalization: Recon auto-finalizes on restart.
  • No downgrade support or behavior blocking: Recon focuses solely on upgrade actions without pre-finalization constraints.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-11465

Developer Design Document

https://docs.google.com/document/d/1GfN-xBvvpAKOMgzoEmf_GOFt-nlyCMu9ZHqj5ug5l3w/edit?addon_store&tab=t.0#heading=h.tfszc17ucbxl

How was this patch tested?

For testing this PR, three example features were introduced into ReconLayoutFeature along with their respective upgrade scripts. These scripts were designed to simply log their execution, allowing us to verify that the upgrade framework works as intended. During Recon startup, the current Metadata Layout Version (MLV) was set lower than the Software Layout Version (SLV). The logs confirmed that each feature's upgrade action executed sequentially, updating the MLV and recording the changes in the RECON_SCHEMA_VERSION table. This successful test demonstrates that the framework correctly finalizes registered features when MLV < SLV.
I'll remove them once the PR is approved by everyone.

Logs showcasing upgrade :-

[main] INFO recon.ReconServer: Creating Recon Schema.
[main] INFO upgrade.ReconLayoutVersionManager: Current MLV: 0. SLV: 3. Checking features for registration...
[main] INFO upgrade.ReconLayoutVersionManager: Feature FEATURE_1 (version 1) is registered for finalization.
[main] INFO upgrade.ReconLayoutVersionManager: Feature FEATURE_2 (version 2) is registered for finalization.
[main] INFO upgrade.ReconLayoutVersionManager: Feature FEATURE_3 (version 3) is registered for finalization.
[main] INFO upgrade.Feature1UpgradeAction: Executing Feature 1 upgrade:
[main] INFO recon.ReconSchemaVersionTableManager: Inserted new schema version '1'.
[main] INFO upgrade.ReconLayoutVersionManager: MLV updated to: 1
[main] INFO upgrade.ReconLayoutVersionManager: Feature 1 finalized successfully.
[main] INFO upgrade.Feature2UpgradeAction: Executing Feature 2 upgrade
[main] INFO recon.ReconSchemaVersionTableManager: Updated schema version to '2'.
[main] INFO upgrade.ReconLayoutVersionManager: MLV updated to: 2
[main] INFO upgrade.ReconLayoutVersionManager: Feature 2 finalized successfully.
[main] INFO upgrade.Feature3UpgradeAction: Executing Feature 3 upgrade
[main] INFO recon.ReconSchemaVersionTableManager: Updated schema version to '3'.
[main] INFO upgrade.ReconLayoutVersionManager: MLV updated to: 3
[main] INFO upgrade.ReconLayoutVersionManager: Feature 3 finalized successfully.

[errose28]

Thanks for working on this @ArafatKhan2198. Can we use the existing concept of "layout features" for Recon the same as they are used on other components instead of introducing a new concept unique to Recon? Recon will need to support downgrades the same as other components, and once we start versioning things it will be easier to let the existing upgrade framework handle that.

[devmadhuu]

Thanks @ArafatKhan2198 for working on this patch. Few comments. Pls check.

[ArafatKhan2198]

@errose28 @sumitagrawl
Could you please go through the description again for this patch.
I have made changes as per our discussion, where we try to bring versioning to entire of recon rather than just the derby tables.

[devmadhuu]

@ArafatKhan2198 , In Recon context, what all MLV will incorporate only Derby tables schema update ? or will it take care recon rocks DB schema changes as well like container DB ?
And SLV is only feature change without any schema changes ? Whether addition of feature will automatically make MLV also eligible for increment in version along with SLV version ?
Can you pls explain ?

[ArafatKhan2198]

@ArafatKhan2198 , In Recon context, what all MLV will incorporate only Derby tables schema update ? or will it take care recon rocks DB schema changes as well like container DB ? And SLV is only feature change without any schema changes ? Whether addition of feature will automatically make MLV also eligible for increment in version along with SLV version ? Can you pls explain ?

Hi @devmadhuu,

In the context of Recon, the MLV (Metadata Layout Version) is not limited to just the Derby DB tables. It encompasses all Recon-specific components, including both Derby tables and RocksDB tables like the container DB. Essentially, MLV manages and tracks any schema changes or modifications in the Recon data structures, ensuring they are all updated consistently.

The SLV (Software Layout Version), on the other hand, represents the software-defined feature set. It covers all new features that may or may not involve schema changes. When a new feature is added that modifies or affects the Recon DB schemas, whether it's RocksDB or Derby DB, the SLV version will get bumped, prompting an MLV increment. This is managed with the help of upgrade scripts during the finalization process.

In summary:

• MLV includes all schema changes across Recon, including both Derby and RocksDB tables.
• SLV pertains to the overall set of Recon features, and adding a feature that involves schema changes will cause the SLV version to be bumped, automatically resulting in an MLV increment through the upgrade scripts.

Let me know if this clarifies your concerns!

[devmadhuu]

Pls fix checkstyle issues.

[devmadhuu]

@ArafatKhan2198 , In Recon context, what all MLV will incorporate only Derby tables schema update ? or will it take care recon rocks DB schema changes as well like container DB ? And SLV is only feature change without any schema changes ? Whether addition of feature will automatically make MLV also eligible for increment in version along with SLV version ? Can you pls explain ?

Hi @devmadhuu,

In the context of Recon, the MLV (Metadata Layout Version) is not limited to just the Derby DB tables. It encompasses all Recon-specific components, including both Derby tables and RocksDB tables like the container DB. Essentially, MLV manages and tracks any schema changes or modifications in the Recon data structures, ensuring they are all updated consistently.

The SLV (Software Layout Version), on the other hand, represents the software-defined feature set. It covers all new features that may or may not involve schema changes. When a new feature is added that modifies or affects the Recon DB schemas, whether it's RocksDB or Derby DB, the SLV version will get bumped, prompting an MLV increment. This is managed with the help of upgrade scripts during the finalization process.

In summary:

* MLV includes all schema changes across Recon, including both Derby and RocksDB tables.

* SLV pertains to the overall set of Recon features, and adding a feature that involves schema changes will cause the SLV version to be bumped, automatically resulting in an MLV increment through the upgrade scripts.

Let me know if this clarifies your concerns!

Pls correct the terms here, looks like you are using both terms for schema layout versioning..

[devmadhuu]

Thanks @ArafatKhan2198 for explaining the patch and motive behind this patch. Overall looks good, just few comments and also pls update the comments in all code to. reflect the SLV meaning as Software Layout Version rather as Schema Layout Version.

As discussed, later once code freezed, you will also remove the dummy created features. Thanks.

[devmadhuu]

Pls correct all comments in whole PR and replace the meaning of SLV (Schema layout version) to (Software Layout version)

[ArafatKhan2198]

Testing out the Dynamic Upgrade action handling for Recon Schema Versioning

---

For testing purposes, I introduced another upgrade action type called TEST_ACTION alongside AUTO_FINALIZE for the feature named FEATURE_3. For this feature, I assigned two upgrade actions—one of type AUTO_FINALIZE and the other of type TEST_ACTION. These are only used for testing and will be removed later.

When I started up Recon, I observed the following log outputs:

[main] INFO upgrade.ReconLayoutVersionManager: Current MLV: 0. SLV: 3. Checking features for registration...
[main] INFO upgrade.ReconLayoutVersionManager: Feature FEATURE_1 (version 1) is registered for finalization.
[main] INFO upgrade.ReconLayoutVersionManager: Feature FEATURE_2 (version 2) is registered for finalization.
[main] INFO upgrade.ReconLayoutVersionManager: Feature FEATURE_3 (version 3) is registered for finalization.
[main] INFO upgrade.Feature1UpgradeActionForAutoFinalise: Executing Feature 1 upgrade script for action type AUTO_FINALIZE...
[main] INFO recon.ReconSchemaVersionTableManager: Inserted new schema version '1'.
[main] INFO upgrade.ReconLayoutVersionManager: MLV updated to: 1
[main] INFO upgrade.ReconLayoutVersionManager: Feature 1 finalized successfully.
[main] INFO upgrade.Feature2UpgradeActionForAutoFinalise: Executing Feature 2 upgrade script for action type AUTO_FINALIZE...
[main] INFO recon.ReconSchemaVersionTableManager: Updated schema version to '2'.
[main] INFO upgrade.ReconLayoutVersionManager: MLV updated to: 2
[main] INFO upgrade.ReconLayoutVersionManager: Feature 2 finalized successfully.
[main] INFO upgrade.Feature3UpgradeActionForAutoFinalise: Executing Feature 3 upgrade script for action type AUTO_FINALIZE...
[main] INFO recon.ReconSchemaVersionTableManager: Updated schema version to '3'.
[main] INFO upgrade.ReconLayoutVersionManager: MLV updated to: 3
[main] INFO upgrade.ReconLayoutVersionManager: Feature 3 finalized successfully.

Currently, for Recon, we've decided that upgrade actions will be autofinalized. This means that only actions marked with the type AUTO_FINALIZE are selected and executed. In the case of FEATURE_3, the upgrade action of type TEST_ACTION was not executed at all. This is because the code is specifically designed to execute only AUTO_FINALIZE upgrade actions, and thus, there is no log output like, "Executing Feature 3 upgrade script for action type TEST_ACTION."

This confirms that annotation-based upgrade action execution is working as expected, as only the appropriate actions (in this case, AUTO_FINALIZE) are executed while others are ignored.

@sumitagrawl @devmadhuu please go through this

[sumitagrawl]

@ArafatKhan2198 Plz check few minor comments

[devmadhuu]

@ArafatKhan2198 , thanks for improving the patch. Few comments. Pls check.

[sumitagrawl]

Give few more comments

[devmadhuu]

Thanks @ArafatKhan2198 . PR changes LGTM now. +1

[sumitagrawl]

LGTM