Expose connection creation form as JSONSchema from sidecar API

[rohitsanj]

Background

The sidecar should return a map of JSON Schemas for the following connection types:

• Direct Connections: This connection type would accept generic configs for establishing connections to a Kafka cluster, as well as an optionally provided Schema Registry cluster.
• Platform/MDS Connections: This connection type would accept configs necessary for discovering resources (Kafka and SR, to begin with) in a Confluent Platform environment using the MDS. (We might rename Platform Connections to MDS Connections since it's technically possible to connect to CP Kafka/SR clusters via Direct Connection given the right set of configs.)

Direct Connections

• kafka:
  • type: map
  • required: true
  • bootstrap_servers:
    • type: string
    • validation: list of hostname:port pairs
  • authentication:
    • type: enum
    • values: [basic, oauthbearer, kerberos, apikey, sasl]
• schema_registry
  • type: map
  • required: false
  • uri
    • type: string
    • validation: must be a URL
  • authentication
    • type: enum
    • values: [basic, oauthbearer, kerberos, apikey, sasl]

MDS Connections

• mds_addresses:
  • type: string
  • required: true
  • validation: list of hostname:port pairs
• authentication
  • required: ??
  • type: enum
  • values: [basic, oauthbearer, kerberos, apikey, sasl]
    TODO: describe configs required for each auth type

[rohitsanj]

Here's a rough draft of the JSON Schema document for Direct Connection:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Direct Kafka Connection Configuration",
  "description": "Configuration for directly connecting to a Kafka cluster with an optional Schema Registry.",
  "type": "object",
  "properties": {
    "kafka": {
      "type": "object",
      "properties": {
        "bootstrap_servers": {
          "type": "string",
          "title": "Bootstrap Servers",
          "description": "Comma-separated list of hostname:port pairs for Kafka."
        },
        "credentials": {
          "type": "object",
          "oneOf": [
            {
              "$ref": "#/definitions/noAuth"
            },
            {
              "$ref": "#/definitions/basicAuth"
            },
            {
              "$ref": "#/definitions/oauthBearer"
            },
            {
              "$ref": "#/definitions/apiKeySecret"
            }
            // Additional authentication methods can be inserted here.
          ]
        }
      },
      "required": ["bootstrap_servers", "credentials"],
      "additionalProperties": false
    },
    "schema_registry": {
      "type": "object",
      "properties": {
        "uri": {
          "type": "string",
          "title": "Schema Registry URI",
          "format": "uri",
          "description": "The URI of the Schema Registry."
        },
        "credentials": {
          "type": "object",
          "oneOf": [
            {
              "$ref": "#/definitions/noAuth"
            },
            {
              "$ref": "#/definitions/basicAuth"
            },
            {
              "$ref": "#/definitions/oauthBearer"
            },
            {
              "$ref": "#/definitions/apiKeySecret"
            }
            // Additional authentication methods can be inserted here for the Schema Registry as well.
          ]
        }
      },
      "required": ["uri", "credentials"],
      "additionalProperties": false
    }
  },
  "required": ["kafka"],
  "additionalProperties": false,
  "definitions": {
    "noAuth": {
      "type": "object",
      "properties": {
        "type": {
          "type": "string",
          "const": "none"
        }
      },
      "required": ["type"],
      "additionalProperties": false
    },
    "basicAuth": {
      "type": "object",
      "properties": {
        "type": {
          "type": "string",
          "const": "basic"
        },
        "username": {
          "type": "string"
        },
        "password": {
          "type": "string"
        }
      },
      "required": ["type", "username", "password"],
      "additionalProperties": false
    },
    "oauthBearer": {
      "type": "object",
      "properties": {
        "type": {
          "type": "string",
          "const": "oauthBearer"
        },
        "token": {
          "type": "string"
        }
      },
      "required": ["type", "token"],
      "additionalProperties": false
    },
    "apiKeySecret": {
      "type": "object",
      "properties": {
        "type": {
          "type": "string",
          "const": "apiKey"
        },
        "key": {
          "type": "string"
        },
        "secret": {
          "type": "string"
        }
      },
      "required": ["type", "key", "secret"],
      "additionalProperties": false
    }
  }
}

And here's a few sample JSONs adhering to the spec above:

1. With basic auth for both Kafka and SR

{
  "kafka": {
    "bootstrap_servers": "kafka1.example.com:9092,kafka2.example.com:9092",
    "credentials": {
      "type": "basic",
      "username": "kafkaUser",
      "password": "kafkaPassword"
    }
  },
  "schema_registry": {
    "uri": "http://schema-registry.example.com",
    "credentials": {
      "type": "basic",
      "username": "registryUser",
      "password": "registryPassword"
    }
  }
}

2. With no auth for Kafka and no SR

{
  "kafka": {
    "bootstrap_servers": "kafka1.example.com:9092,kafka2.example.com:9092",
    "credentials": {
      "type": "none"
    }
}

[rhauch]

Nice! WDYT about a way for SR to say "Same credentials as Kafka Cluster"? Do we bother?

[rohitsanj]

Yeah, that'd be nice. Although, not sure if we can do much in that regard with the JSON Schema itself, feels like it'd be something we codify in the dynamic form generation. Worth mentioning here confluentinc/vscode#415.