Extract connector specifications into connector.yml

[lovromazgon]

In this discussion, we're proposing a change in the connector development experience. Currently the specifications of a connector plugin are written in pure code. There are a few drawbacks to that:

• You have to run the connector plugin to fetch the specifications.
• If you want to parse the specifications without running the connector, you have to parse code, which makes it hard to do across different languages.
• Fetching the specifications remotely (without checking out the repository or building the connector) is not possible. This is important for generating documentation for all supported connectors on conduit.io.

To address these pain points we propose to introduce connector.yml, a file that will by convention live at the root of each connector repository and will contain the connector specification. There will be tooling around keeping the file up to date, using it to generate parts of the readme, using it to efficiently generate consistent documentation for connectors on conduit.io, to serve specifications in the connector (same as connectors already do) without introducing breaking changes.

1. Specgen

Specgen replaces paramgen. It takes the source config and destination config structs, parses the parameters (same as paramgen) and outputs connector.yml instead of Go code.

It's used in a go:generate statement, so a CI action can make sure connector.yml is up to date when running make generate.

The preferred place for this statement is in connector.go.

//go:generate specgen -source.config SourceConfig -destination.config DestinationConfig -o connector.yml

2. connector.yml

The generated file connector.yml replaces spec.go entirely. It contains the whole specifications and is language agnostic.

The only caveat is the version field, which contains the last released version. It is bumped only when releasing the next version. The very update of the version actually triggers the release (see version handling).

version: 1.0
connector:
  name: file
  author: Meroxa, Inc.
  version: v0.6.0
  summary: A file source and destination plugin for Conduit.
  # The description can contain markdown.
  description: |
    The file source allows you to listen to a local file and detect new lines
    added to it. Each change will create a new record. The destination allows
    you to write record payloads to a destination file, each new record payload
    is appended to the file in a new line.
  # ------------------------ GENERATED ------------------------
  source:
    parameters:
      - name: path
        type: string
        description: The path to the file to read from.
        default:
        validations:
          - type: required
  destination:
    parameters:
      - name: path
        type: string
        description: The path to the file to write to.
        default:
        validations:
          - type: required

3. YAML specification parsing

The generated connector.yml is embedded in the connector binary and is used to create the specifications of the connector. The Source and Destination structs don't have to implement Parameters anymore, although the functions are kept for backwards compatibility.

Notice gitRef, which is populated during the build process with the current ref. sdk.YAMLSpecification uses gitRef to update the version in the specifications:

• If gitRef matches field connector.version from connector.yml, the version is left as is (only true if the checked out ref is tagged, i.e. a release).
• If gitRef is populated, but does not match connector.version from connector.yml, the version is updated to gitRef (will be true during development).
• If gitRef is empty, the version gets the postfix -(devel).

// gitRef is set during the build process (i.e. the Makefile).
var gitRef string

//go:embed connector.yml
var connectorYaml string

var Connector = sdk.Connector{
  NewSpecification: sdk.YAMLSpecification(connectorYaml, gitRef),
  NewSource:        NewSource,
  NewDestination:   NewDestination,
}

4. Version handling

We need to add some safeguards to ensure the connector.version field in connector.yml contains the correct version and matches the git tag. The idea is to use two GitHub actions to automate this:

• A GitHub action is triggered BEFORE release, checks if the pushed tag matches the tag in connector.yml. If it does NOT match, it deletes the tag and fails the action, preventing the release.
• Releasing happens by opening a PR that updates connector.version in connector.yml manually. If needed, we can provide a script that bumps it for us.
• A GitHub action that detects any commit to main which updates the version in connector.yml. If it detects one, it pushes a tag with the updated version, triggering a release.

The result is a connector.yml that contains the correct latest released version which matches the last git tag.

5. Readmegen

Readmegen uses connector.yml as input to populate sections in the readme. It should be run as part of make generate, to allow the CI action to ensure the readme is up-to-date.

readmegen:
    readmegen -c connector.yml -f README.md -w

generate: readmegen
    go generate ./...

6. Generate connector documentation on conduit.io

Using the introduced connector.yml we can then generate a documentation page for each connector on conduit.io.

In the same way we currently generate the list of connectors, we can additionally fetch connector.yml using the GitHub API for a specific release:

gh api \
  -H "Accept: application/vnd.github.raw+json" \
  -H "X-GitHub-Api-Version: 2022-11-28" \
  "/repos/conduitio/conduit-connector-file/contents/connector.yml?ref=v0.6.0"

Then we can supply our own template that creates consistent documentation pages for all connectors.

readmegen -c connector.yml -f doc-template.md > file-connector-doc.md

Alternatively, we can fetch README.md directly from the connector repo, as it might provide additional information. Then again, if we do that, we can't ensure a consistent look and feel of the documentation for all connectors. Also, we rely on the connector developer to have all actions in place to ensure the readme is up-to-date. That's why it might be better to generate the doc and refer the user to the repository for additional information.

7. Plugin binary commands

This is somewhat unrelated to the changes above, but it would be useful if the plugin binaries would support a few additional commands. We can implement this transparently in sdk.Serve and it will be applied to all connectors.

I propose two commands:

• connector reference would print the full reference for this specific connector.

  ./conduit-connector-file reference
  
  standalone:[email protected]

• connector specs (or connector specifications) prints the YAML specifications of the connector.

  ./conduit-connector-file specs
  
  version: 1.0
  connector:
    name: file
    ...