Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ENH] schema - specify what is valid at the root of a dataset #1108

Open
wants to merge 15 commits into
base: master
Choose a base branch
from

Conversation

rwblair
Copy link
Member

@rwblair rwblair commented May 26, 2022

prior art:
#884
https://github.com/ANCPLabOldenburg/ancp-bids/blob/main/ancpbids/data/bids_graph_schema.yaml

Problems

  • Current schema does not explicitly have a way of what is a directory as opposed to a file.
  • inconsistent ways of describing filenames. top_level_files.yaml and associated_data.yaml use the key of the object to specify the filename (excluding extension) while file names described in datatypes/*.yaml use a combination of entities+suffix+extension to describe a filename

new keys used in rules to help resolve issues:

  • type: directory
    we currently have no way of saying the thing I'm describing is a directory. If omitted we could assume its a file, or we could make every file use type: file
  • children:
    this would specify what valid file and directory names inside a given directory are
  • literal: Decided to use pattern to cover this use case.
    some files in datatypes are simple singleword suffixes with an extension. When interpreting entity-suffix-extension pattern rules we use the underscore as the split character separating entities from each other and entities from the suffix.
    There are files like 'dataset_description.json' whose preextension string have an underscore, so using suffix-extension with no entity would need to be understood as 'dont split on _'
    literal gets around that by saying hey just look for this exact thing, forget about entities or suffixes and splitting.
  • pattern:
    pipelines can have any name. would be interpreted as a regex to match against the directory or filename literal and pattern could be merged, regex matches literals just fine.

A file or directory name could then be described with one of the following patterns:

  • entities + suffix + extensions
  • suffix + extensions
  • literal
  • literal + extensions
  • pattern
  • pattern + extensions

Another issue maybe, not for this PR, is how implicit references to other parts of the schema are handled, they're context sensitive. If I have an entry under an entities key I know to go look in objects.entities. Here with children I've referenced keys that appear in the same file without a $ref tag.

@rwblair rwblair changed the title [ENH] specify schem [ENH] schema - specify what is valid at the root of a dataset May 26, 2022
@rwblair
Copy link
Member Author

rwblair commented May 26, 2022

All rules describing valid filenames could also be moved into a single directory to help keep everything together.

@rwblair rwblair marked this pull request as ready for review July 7, 2022 19:30
@rwblair rwblair requested a review from tsalo as a code owner July 7, 2022 19:30
@Remi-Gau Remi-Gau added the schema Issues related to the YAML schema representation of the specification. Patch version release. label Dec 22, 2023
@sappelhoff sappelhoff requested a review from erdalkaraca as a code owner June 7, 2024 13:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
schema Issues related to the YAML schema representation of the specification. Patch version release.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants