ENH: Add num_columns, num_rows to table spec

[mferrera]

Resolves #768

[mferrera]

I forget why I type annotated this, and why it's in this PR, but let's call it a freebie

[mferrera]

I am worried about retrospective validation. Do these fields need to be optional for backward compatibility?

[tnatt]

I am worried about retrospective validation. Do these fields need to be optional for backward compatibility?

This is difficult.. how well is the sumo schema aligned with the komodo version? there is no versioning of the schema atm right?
I guess the safest is to have them optional for a transition period, in case users lock their komodo version?

[mferrera]

I think we just need to tackle versioning sooner rather than later. We have already made far too many changes and called it version 0.8.0. The more changes we implement the higher the risk we cause service interruptions for users. We need to get the fundamentals of deployments right otherwise we are going to build ourselves into a hole we can't get out of.

[perolavsvendsen]

I think we just need to tackle versioning sooner rather than later. We have already made far too many changes and called it version 0.8.0. The more changes we implement the higher the risk we cause service interruptions for users. We need to get the fundamentals of deployments right otherwise we are going to build ourselves into a hole we can't get out of.

Yes.

(Missing) schema versioning has been a headache for years. We have discussed linking it to the fmu-dataio version, we have discussed how to migrate between schemas, etc etc. For now, the (not really a) solution has been to make sure we are backwards compatible at all times, and just stay on version 0.8.0. The story behind that is really just that the schema used on JS was on version 0.4 when development of fmu-dataio started. Then it was incremented a few times, until it sort of just froze on 0.8.0 🤷‍♂️

However, it needs to be extremely well aligned across fmu-dataio and Sumo and all consumers, including Webviz, REP, DynaGeo etc etc would have to also deal with it somehow, either explicitly or implicitly through some sort of abstraction. This also links a little bit to the discussions around products - today consumers will speak directly to the (meta)data via Elasticsearch.

So yeah, this is fun topic :)

[mferrera]

Updated these fields to be optional for now. They don't need to be, because the user input isn't required to populate them, but it will maintain backward compatibility.

[tnatt]

🚀