Versioning philosophy

[MikeNeilson]

Okay, so everyone (me too, just worried about the security people) wants a simple "WTF version is this instance", I want to switch to calendar versioning (CalVer) and provide such an endpoint. However, beyond that I think we need to better formalize what our total versioning is since there's definitely been some confusion on that point.

The following will likely read as a manifesto so bear that in mind as at this time I'm not dictating exactly this as the solution but soliciting feedback. So please argue with me, I will attempt to bring references, you should do the same.

Some of this is already stated on the wiki page, but it keeps coming up in various PRs and issues so I think good to bring up a more general case and formalize the docs in a way easier for others to common and propose changes.

There may be differences from https://github.com/USACE/cwms-data-api/wiki/API-Design, That was written a while ago and we have some experience with the API and it's usage now. Anything we decide in this discussion will supersede that document and that page will either be deleted or updated.

---

The general philosophy of CDA

In general

1. The API itself is not versioned. We will use CalVer on the code and provide this on request as in some cases it will be easier to clients to make certain determinations.
2. A URL or URI identifies a resource (Location, timeseries, etc)
3. the authority identifies a CDA instance
4. The Path identifies the type of resource
5. The query parameters identify which of that type is desired
6. a MIME TYPE (Content-Type) identifies the format of a resource, e.g. XML, JSON, JSON with X,Y,Z features RFC2025

Identification of data

1. A URL with names expanded out (e.g. human friendly) is the canonical identification of the resource. e.g /rating/South%20Diversion%Canal%.Stage;Flow.Prod along with any query parameters
2. We will support a more machine friendly ways of accessing data, e.g. something like /rating/8d7197ee-7670-429f-a57e-38abb9c8c4ea. OR some sort of SLUG that internal can used.

Underlying data stores

As CDA is intended to be the primary way into data, it is connected to a database. It is not a goal or requirement of the API to directly map end points to tables, procedures, or views. It is acceptable to do so ; however, the general usage of data expected should be considered and provided in a friendly way that could be rendered as-is in a web page.

Without good reason, we should not expose relational database design constructs through the API. The API exposes Data, not Tables.

Primary target of operations

Usage within websites and system grabbing just data are the primary user base. The design and documentation should focus on those usages, the API itself while providing a simple swagger-ui is NOT intended to be consumed manually. While we should strive to make it easier to do so, it's not the job of the API but systems that use the API on those users behalf.

Versioning of data variable shape data

As stated the data, by content-types, are versioned. In the past there was some severe confusion on this part and it was treated as anything new was "version=2" in the content-type. To allow this design but reduce confusion going forward

1. The initial version of a data set SHALL be the date it was finalized (e.g. PR about to be merged)
2. IF it is the first version of this data the plain content-type "e.g. application/json" will point to this data
3. IF is it not the version version of this data, it will be discussed and announced when it becomes the new default data.
4. Downstream systems SHOULD use the specific version regardless of when implemented, and this behavior should be well documented
5. If a given data set includes definitions of its shape within the type there should be sufficient documentation for downstream developers to properly account for any changes over time. (See our TimeSeries type and discussions within Fixes #634 - Implemented data entry date option for TS data retrieval #927 .

Database interaction design

1. Open a few connections as possible for a given set of operations.
2. If at all possible open a single connection for all operations during a request.
3. If that is not possible, document it for future research.

Searchability (this one is really just a question, not an assertion.)

We need to decide on a single "CATALOG" end point, using the plain endpoint as a catalog e.g. /timeseries, or a catalog for each data type, e.g. /timeseries/catalog

While the current Catalog of time series and locations has query parameters than overlap well, I suspect it will get rather confusing if we add anymore.

@MikeNeilson is in favor of the <datatype>/catalog concept as it makes it explicit.
Alternativaely current /catalog/<datatype> uses a PATH parameter and that could instead just be a path to a separate endpoint. That would group all of the CATALOG endpoints in swagger while allowing them to list and document their own parameters.

Deployment philosophy

CDA should be continuously deployed (at a minimum into test environments) so that all code is affected whenever important changes are made while also not stopping development. As such any endpoint can, and new endpoint must, determine by some method if they can reasonably execute and if not return an appropriate "operation not supported" response to the user.

The result of this check may be cached. A negative results should be cached no more than 1 hour.
Source our eventual version endpoint generate and provide this list? I believe we've all said at one point it's important for systems to be able to determine what's available and give it's a publicly documented and open source API it's not like we're really giving anything away from a security standpoint.

Follow up

After probably both way to much and not enough discussion (RMA folks don't go overboard on the time), I will document the consensus. Probably in a similar, if not identical, style as cwms-python.

Will also separately document anything that still has clear disagreement and requires follow up discussion or more information. Was looking up some of the ADR type docs for OpenDCS and that's not uncommon and it would be could to leave such these in the same place as it would make it easier to move to accepted.

@jbkolze @krowvin @DanielTOsborne @adamkorynta @rma-bryson @rma-psmorris @rma-rripken @zack-rma
@ktarbet @mdstanfill @katfeingold @adamscarberry @jeffsuperglide @rgoss @oskarhurst @jvanaalsburg

Please send anyone who may be interested a link to this to comment on. Above are just the names that have either previously comment on such, may be interested, or whose opinion I value (okay, I value them all. Favor, perhaps?... we all have our biases.)