Breaking changes to default settings should behave like breaking changes to the API

[AmiStrn]

Is your feature request related to a problem? Please describe.
When upgrading to newer versions defaults may change (e.g. the default of max_open_scroll_context changed from unlimited in ES 6.8 to 500 in OpenSearch 1.X). The upgrade may go smoothly and then the cluster displays odd behavior due to the defaults changing. There is no way to ensure ahead of time except to validate one setting at a time against an ever changing cluster state.

Describe the solution you'd like
Two possibilities:

• Two-phase changes - the first major version contains the new default in WARNING mode, and the next enforces it. the warning mode produces logs explaining the thresholds crossed and suggests documentation to improve performance or at least suggest a given setting threshold has been crossed.
• The second option is an upgrade tool that runs for a while on the cluster analyzing if it is ready to be upgraded, and expects no thresholds to be broken in the new version.

Describe alternatives you've considered
reading the documentation and gathering metrics for weeks before an upgrade. not fun.

Additional context
relates to #3757

[dblock]

OpenSearch follows semver since 1.0, so we already treat default settings changes as breaking changes and only do that in a major release. ES 6.x -> OpenSearch 1.0 was before that time, but 7.10 = 1.0 for semver purposes. For major versions we document breaking changes. I'll close this, but do reopen if you run into a specific example where we did the wrong thing in OpenSearch, please.

If you have a list of all such breakages between ES and OpenSearch I definitely think #3757 would appreciate that list.

[AmiStrn]

The issue is not that the default changes be listed, and documented as breaking, but that it should have two phases: A warning on changed setting phase and then a breaking change. Each in separate major versions (like api changes).

[dblock]

You mean code should do that? Reopened.