Does this specification allows for one to specify the upstream system source for the data or just the delivery server?

[taksqth]

Not sure if I missed it in the specification, but it is common to have an upstream system such as MongoDB and the deliver contract referring to a middle layer such as S3. From what I understood, server refers to the later, but it is useful to have more information regarding the true upstream original source. How would I, if it is possible, annotate this in the current schema?

[jochenchrist]

You can use the lineage field for upstream information: https://datacontract.com/#lineage-object

Example:

models:
  orders:
    fields:
      order_id:
        type: string
        lineage:
          inputFields:
            - namespace: my-mongodb
              name: checkout_db.orders
              field: order_id
              transformations:
                - type: DIRECT
                  subtype: IDENTITY
                  description: The order ID from the checkout order
            - namespace: my-mongodb
              name: checkout_db.orders
              field: order_timestamp
                - type: INDIRECT
                  subtype: SORT
      customer_email_address_hash:
        type: string
        lineage:
          inputFields:
            - namespace: my-mongodb
              name: checkout_db.orders
              field: email_address
              transformations:
                - type: DIRECT
                  subtype: Transformation
                  description: The email address from the checkout order, hashed with SHA-256
                  masking: true

Hope this helps.

[taksqth]

Hi @jochenchrist thanks for the reply, sorry it took a while to get back.

This does work, although it does lead to some repetition. In most cases there's a N:1 relation between the delivery servers and the upstream server, so having the upstream information in each field sounds a bit redundant specially when there are no transformations being employed. In my usecase, I'll probably omit this from the contract and note it elsewhere (maybe in the architecture diagram).

There's one other bit that I'm confused on, which is the contract granularity. I don't know if I missed it, but it didn't seem to me like the specification assumed any sort of underlying data model, although the example can be interpreted as a single model for 1 fact (order) + 1 dimension (line_item) table. In cases where I have one server with N models, possibly multiple facts, should I split the contracts? At least in the current format, it doesn't seems like it is possible to work with model-scoped service levels, which would make sense if I had, for example, two fact tables for different order flows.