Better handling of custom bids entities

[pvandyken]

With the release of #136, Snakebids is now at least able to receive custom pybids config files for consumption by pybids. Unfortunately, a very serious limitation remains preventing the use of entities whose name (e.g. "foobarspam") is different from their tag (as found in the file name, e.g. (_fbs-{foobarspam}). The only way currently to create an equivalency between the two is to update the bids_tags.json, which is not a publicly accessible file.

A couple of options come to mind to resolve this, and I'd be curious if there's any additional solutions. @akhanf @kaitj @tkkuehn

Allow custom bids_tags.json

We've actually discussed this idea before and decided not to. I forget what the exact reasons were: I think it partially came down to not wanting to make a wild west of bids tags (although that concern may have been directed at letting the file determine the order of the tags in the path). It's still a possibility though. Users can create a custom bids_tags.json file and point generate_inputs() to the file.

I think a big concern here is the need to maintain two separate files, both of which do the same thing (an issue which has been raised before).

Attempt parsing of pybids Entity patterns in generate_inputs()

The layout.get_entities() method from pybids already gives us the long name of the entity and the pattern. We could attempt to parse that pattern to obtain the tag name. This would free us from the use of bids_tags.json and would support any arbitrary pybids config file. Sort of. The issue here is that while entities are formally standardized, we can never guarantee the pattern will take the exact form we expect, so as soon as any custom tags are used, we'll have a hard time ensuring consistent outcomes. If we wanted to go this route, I think our best bet would be to petition pybids to add tag as part of their entity spec, but I'm not sure that's really a fair request, especially since not every member has a tag (e.g. datatype, suffix, extension, prefix, etc)

Make bids() aware of the pybids config file

Here, we allow bids() to use the match patterns built into the pybids config file, following back on our generic patterns whenever the tag is unrecognized. If built correctly, this would allow users to create pybids_config.json files that capture their entire range of inputs and outputs, solving all the ordering and entity definition issues we've had. I think this is doable; the greatest hurdle is how to make bids() aware of the config file.

We obviously could just make the path another parameter, but you'd have to pass it every time you call bids(), which seems wordy and unnecessary.

One approach is to do it Snakeboost style: make bids() a class that's first initialized with settings like config_file_path, and then remains callable thereafter. This would be breaking, although we could do a phased release (start by having both the classic bids() function and a Bidsclass, then deprecate thebids()` function, etc).

Another is to have some global registration object:

from snakebids import config
config.set_pybids_config("path/to.json")

These patterns can work well if the app is well organized (e.g. the python logging library works this way). Of course, order matters. The call would have to be e.g. in the setup.smk. If you get the order wrong, accidentally calling bids() before setting config, the resulting errors would be subtle (bids would just output paths with different names than expected). We could raise a warning, but this pybids_config setting should be totally optional, so I don't think that would be appropriate.

Allow the defining of custom wildcard entities in config.yml

We allow an expanded syntax to wildcard definitions:

datatype:
  wildcards:
    foobarspam:
        name: foobarspam # The name as written in the pybids config
        tag: fbs         # The tag to use in the path

One nice thing here is that this could negate the need to provide custom pybids_config.json files. Currently, we check if an entity is defined in layout.get().get_entities() for filtering purposes, but we don't use any other pybids machinery. With that expanded syntax, we have everything we need to parse the path ourselves, exactly how we are currently doing. Obviously, the user will be limited strictly to _tag-{wildcard} patterns.

On the other hand, this creates a divide between wildcards and filters, which will still use pybids machinery. So any filters would still have to be pybids-valid, or defined in a pybids config.

[pvandyken]

I should say another option is to do nothing: just make it a known limitation that a custom entity long name must be the same as the tag name

[pvandyken]

The point made here will be resolved with #245