Data stewardship, publication formats and access methods

[duncandewhurst]

Design choices in the development of the Open Fibre Data Standard must be informed by an understanding of how data will be collected and shared and the publication formats and access methods required to meet user needs.

We have prepared a consultation document with the aim of gathering feedback from stakeholder and of establishing a shared understanding of the proposed approach to these areas.

You can provide feedback by commenting and suggesting edits directly in the document or by commenting on this Github issue.

[duncandewhurst]

@stevesong raised a question in #14 on which it would be great to hear input from other stakeholders:

The number one question for me is what is the CSV use case? Perhaps if the records are capturing the kilometre length of links, it would be useful to open in a spreadsheet. Other scenarios?

I'll share my feedback in a separate comment.

[duncandewhurst]

Good question. Tabular data was the only publication format preference expressed in the stakeholder interviews, but we didn't hear about specific use cases for it. The opinions expressed were that publishing and using data on nodes and links (without detailed routes of links) would be most straightforward in a tabular format. The supply side research also surfaced several examples of tabular publications so a tabular format may be more familiar/achievable for some publishers.

That said, many of the use cases identified in the demand side research were concerned with the detailed routes of links, for which JSON/GeoJSON seems better suited.

In general, if the only formats are JSON/GeoJSON, then users interested only in non-spatial data elements (like link length, investors, status and dates of phases etc.) would need to be skilled in processing JSON data or would need to load the data into a GIS tool to do anything with it, which is a potential barrier to use. CSV data is much easier to work with for users who are only skilled in using spreadsheets or databases.

[stevesong]

As I think about it further, there are probably many instances where tabular data is desirable. I can imagine a spreadsheet looking at installation or upgrade or maintenance or repair status of fibre links. There are no doubt many more examples, although it would be good to hear from someone more directly in the front lines of managing this kind of data.

[lgs85]

At our most recent technical steering group meeting, we had a useful discussion about file formats. Specifically, we discussed the possibility of supporting protobuf format in the standard. We'll post some thoughts on this here, and comments are very welcome.

[lgs85]

We've discussed amongst ODSC colleagues, and think that the issues underlying the above can be addressed through improved guidance. Specifically, there are two concerns that we need to address:

Concerns about large JSON files:

• How to publish large datasets is already addressed by the standard's guidance on the use of compression for bulk files, API endpoint design and support for streaming.
• How to load large datasets onto a map is something to be addressed on a case by case basis by application and web developers, i.e. data users. The typical approach would be to load the data into a database/web mapping service and to only load/render the relevant points (the ones that are viewable at that particular scale and within the viewport) in the client's browser. We can consider adding a section on mapping large datasets to the data use guidance.

Concerns about restricting the standard to particular formats:

• Allowing implementers to publish data in any format creates significant extra work for data users whereas specifying the publication formats that implementers must use makes it possible to develop reusable tools and methodologies for working with the data. This is an example of how developing a standard reduces the total effort required for data conversion since conversion only needs to happen once at the publisher's end rather than many times for each data user.
• If none of the standard publication formats are appropriate for an implementer's internal needs, they are of course free to use their own publication formats 'inspired' by the field names, structures and codelists in the standard. However, implementers should publish data for external use in the standard publication formats for the reasons expressed in the previous bullet points. We can consider adding a section on non-standard publication formats to the data publication guidance.

[lgs85]

At our most recent technical steering group meeting, we had an interesting and prolonged discussion about publication formats. In our consultation document we propose that OFDS follows a decentralised publication approach. The main reason for this is that:

Opting for a decentralised publication model reduces the technical and administrative overhead for the initiative since there is no requirement to develop systems and processes for collecting and publishing data centrally. It also removes the risk of a single point of failure for publication.

We note that there are risks with this approach, especially around access to data and data quality. We can use this space to further discuss this issue if required.