Artifacts

Each bundle author is responsible for creating a artifact JSON Schema if their output artifacts are of a new type.

E.g.: If we don't have an AWS VPC type, the aws-vpc author needs to create that type.

Artifact definitions should go through a peer review as they are a critical abstraction of MD.

Any cloud type should be composed of an ID type for that cloud. This type is used across all resources for a particular cloud (GCP, AWS, k8s, Azure).

E.g.: AWS VPC should be composed of an AWS ARN type.

Never $ref a remote type, always include them in this repo so changes are explicit and reviewable via PR. Futher $refs are hydrated into the artifact definition and bundles when published as to not introduce side effects by remote refs being changed.

• ./artifact - artifact definitions for Massdriver
• ./specs - shared specifications for common capabilities provided by an artifact. E.g.: CloudSQL and RDS share the "RDBMS" spec.
  • Specs are also 'non-sensitive' information, thus can be used for filtering and searching in massdriver.
• ./types - simpler common types that can be reused by many artifacts. Types themselves are not artifact definitions.

Development

Install the pre-commit to ensure our json is pretty and our json schemas valid in tighter development cycles.

pre-commit install

Local Server

You can run make local.server to run a local nginx server which hosts the artifacts in a way that the Massdriver CLI can access them. This is convenient for local development of bundles using local artifact definitions.

As you make changes to artifacts or types, you can run make local.update to regenerate the definitions hosted by the local server. If you set the MASSDRIVER_URL environment variable to http://localhost:8081 then the CLI will pull artifact definitions from this local server instead of the Massdriver API. Run make local.shutdown when you are done to shutdown the development server.

Artifacts

Artifacts define connectable pieces of infrastructure in Massdriver. Artifacts that strictly represent infrastructure must have the format (omit fields if not applicable):

{
  "data": {
    "infrastructure": { // REQUIRED
      "$ref": "../types/aws-infrastructure-eks.json"
    },
    "security": { // REQUIRED if applicable
      "policies": { // IAM policies this bundle created
        "read": "my_policy_foo",
        "write": "my_policy_foo2"
      },
      "groups": {
        // Security groups this bundle created, downstream bundles will attach and set up their own security group rules.
        // AWS Example
        // service named => {arn, service port, service protocol}
        "postgresql": {
          "arn": "arn:securitygroup:foo"
          "port": 5432,
          "protocol": "tcp"
        }
      },
    },
    "auditing": {
      // TBD:
    }
  },
  "specs": {
    // SPECS HERE
  }
}

Additionally, bundles that cross into the application plane and generate a credential must include an authentication object. In this case an additional artifact should be emitted in a cloud-agnostic manner if applicable.

Example:

AWS EKS generates a cluster admin credential for Kubernetes. Applications deploy to kubernetes, not EKS.

The AWS EKS bundle should emit two artifacts:

• aws-eks-cluster - This is used to connect infrastructure that does not need k8s credentials
• k8s-cluster - This is used to connect applications that do need credentials

The k8s-cluster's data block should be identical to the EKS's data block with the additional of an authentication field.

When creating these artifacts in terraform a local block should be used to eliminate typos between the two artifacts.

{
  "data": {
    "infrastructure": { // REQUIRED
      "$ref": "../types/aws-infrastructure-eks.json"
    },
    "security": { // REQUIRED if applicable
      "policies": { // IAM policies this bundle created
        "read": "my_policy_foo",
        "write": "my_policy_foo2"
      },
      "groups": {
        // Security groups this bundle created
      },
    },
    "auditing": {
      // TBD:
    }
    "authentication": { // REQUIRED if auth is created
      "token": "foo"
    }
  },
  "specs": {
    // SPECS HERE
  }
}

Artifacts may only forward fields from previous artifacts if the field was immutable. This must only be used if necessary.

• OK to forward example: AWS VPC has a region field, bundles downstream from the VPC may forward the region

• NOT OK to forward example: AWS Pub/Sub fanout creates an IAM Policy document foo, downstream bundles may use foo but must not forward it, as the policy's ARN may change if the bundle were to switch which Pub/Sub subscription it is connected to.

types

Types are a way to create composable infrastructure that is consistent across bundles.

• Types must be scoped by their 'cloud' if cloud specific. e.g.: aws-, gcp-, k8s-
• Types that are not cloud specific, must not be scoped. e.g.: cidr
• Types may be scalar definitions designed for reuse of descriptions, validations, etc. e.g.: cidr.json defines CIDR ranges
• Types must be sub-scoped if the represent a reusable object in artifacts.
  • e.g.: aws-infrastructure- should be used for configurations placed under data.infrastructure in AWS artifacts
  • e.g.: gcp-security- should be used for configurations placed under data.security in GCP artifacts

$md config

There is an top level $md field on some artifact definitions. These allow us to control how the front end treats them.

{
  "$md": {
    "access": "private",
    "defaultTargetConnectionGroup": "credentials",
    "defaultTargetConnectionGroupLabel": "GCP Service Account",
    "importing": {
      "fileUploadType": "json",
      "fileUploadArtifactDataPath": ["data"],
      "group": "authentication"
    }
  }
}

{
  "$md": {
    "name": "the-artifact-name",
    "access": "public",
    "defaultTargetConnectionGroup": "credentials",
    "defaultTargetConnectionGroupLabel": "Kubernetes",
    "importing": {
      "fileUploadType": "yaml",
      "fileUploadArtifactDataPath": ["data", "authentication"],
      "group": "authentication"
    }
  }
}

• name - the name of the artifact without the organizational scope. This should be a URL safe name consisting of letters, numbers, and hyphens. It must be unique to the org it is published under.
• access - specifies the access level of the artifact definition. public is readable by any Massdriver account, private is readable only by the organization that publishes the artifact definition. Defaults to private if unspecified.
• defaultTargetConnectionGroup - allows the artifacts of this type to be set as defaults for a Target, by omitting, it disables this artifact type as defaultable.
• defaultTargetConnectionGroupLabel - is the label to put on the section header for listing these types of artifacts
• importing.fileUploadType - allows files to be uploaded when importing an artifact. This requires that the artifact has the same structure as the file type. Generally only applicable to authentication
• importing.group - the group in onboarding and importing frontend that this artifact definition form should be grouped under.
• importing.fileUploadArtifactDataPath - the json key path to store the deserialized file into. Should be ["data"] if not present.
• export - different file formats of the artifact that can be downloaded
  • downloadButtonText - download button text
  • templateLang - the template to render the artifact in
  • fileFormat - the template format to use (currently only liquid is supported)
  • template - the template