[PROPOSAL] Error or warn on invalid options when sending requests across multiple versions of OpenSearch

[dblock]

What/Why

What are you proposing?

Consider the following example sent to OpenSearch Serverless.

PUT _index_template/return-document-index-template
{
  "index_patterns": [
    "return-document-index-*"
  ],
  "template": {
    "settings": {
      "number_of_shards": 10,
      "number_of_replicas": 1
    },

The response is

  "index_templates": [
    {
      "name": "return-document-index-template",
      "index_template": {
        "index_patterns": [
          "return-document-index-*"
        ],
        "template": {
          "settings": {},
          "mappings": {

In this repsonse the setting part is blank.

The same response from Managed OpenSearch.

{
  "index_templates" : [
    {
      "name" : "return-document-index-template",
      "index_template" : {
        "index_patterns" : [
          "return-document-index-*"
        ],
        "template" : {
          "settings" : {
            "index" : {
              "number_of_shards" : "10",
              "number_of_replicas" : "1"
            }
          },

In OpenSearch Serverless, you don’t specify the shards (primary and replicas). The service takes care of that for you. Any tentative to set this explicitly will be disregarded.

The proposal is to give users the option to error or warn on invalid options being supplied to various APIs across multiple versions of OpenSearch. The client should be negotiating a schema version first, then the client would check the request payload and either fail or warn on differences.

What users have asked for this feature?

This came from early experiments with OpenSearch Serverless.

What problems are you trying to solve?

• Callers want to be able to write the same code that works against multiple versions of OpenSearch.
• However, callers are surprised by the fact that a particular version of OpenSearch ignores some fields.
• However, callers want predictable behavior in which they know that a certain option is going to be applied.

What is the developer experience going to be?

• Clients can configure behavior when server schema is different from request schema. It can be an error, or a warning, or something custom depending on the situation (e.g. allow unsupported options in index operations vs. I send a request with data that will be ignored).

What is the user experience going to be?

The goal is that applications can be written once, and work against as many versions of OpenSearch as possible without any changes, yet behavior is predictable on the edges.

Why should it be built? Any reason not to?

Servers could fail when the client sends data that the server doesn't support. However that makes it difficult to build applications once and make them work against multiple versions of OpenSearch. Thus, Serverless is a subset of the API for that reason and safely ignores cluster options because those are very specific to OSS or Managed OpenSearch. In general, stronger checks are rarely implemented in HTTP protocols. You can append &x=y to about anything, post a form with random fields, or send JSON that has unsupported options, to most APIs.

Thus, while there are pros and cons for failing server-side requests when unsupported options are specified from the clients POV we have to work with existing servers that are unlikely to change behavior.

What will it take to execute?

• When generating schemas via [PROPOSAL] Use API specification to generate clients #19, we will need multiple versions, or annotation for what's supported in which server environment.

[sharp-pixel]

It looks like we need to consider API versioning and/or vendor MIME types to distinguish flavors or evolution for either request or response format.