[SCHEMA] bep012 functional derivatives

[rwblair]

My attempt at schematizing bep012. New in this PR, regex based column names.

[rwblair]

Can't figure out the yamllint error. That line looks a lot like other checks with includes statements

[tsalo]

The new suffixes need to be defined, and I noticed a few typos, but other than that it looks pretty comprehensive.

The three tough things about this BEP are:

1. Columns that use regular expressions. Do we need a new object field for these (e.g., pattern), or is name fine?
2. One-to-many mapping for columns that use regular expressions. Do we need a new object field for these, or can we assume that any regex-based column can appear multiple times in the same TSV file?
3. The new column suffixes that can be applied additively. Do we need a new object type for these? They're not really columns, and they can be applied to a wide range of existing columns.

EDIT: Can you also update your branch to pull in the recent test changes?

[tsalo]

Suggested change

	- ".nii"
	- ".nii.gz"
	- $ref: rules.datatypes.func.func.extensions

Would this work?

[effigies]

I'm not sure if it will work, it doesn't seem to cost much to be explicit. Either way, a thing to note is that it's $ref, not _ref.

[tsalo]

Per our discussion today, I think this may need a level_addendum. I assume we'll want a more conventional metadata table instead of the current one, which has a "Required for suffix" column instead of the typical "Requirement Level" column.

[tsalo]

These new regex-based names are new. I think having regex-based names fits, but one other novel aspect is that the columns will have a one-to-many relationship. All other columns and metadata fields have had one-to-one mappings so far (one object definition for each column/field). Do we need an additional field in the object schema to account for this?

[effigies]

Missing datatype: func here and elsewhere.

[effigies]

I think we can just do this a a single table:

Suggested change

	BandpassFilterOpt:
	selectors:
	- dataset.DatasetType == "derivative"
	- '!(["alff", "falff"].includes(suffix))'
	fields:
	BandpassFilter: optional
	BandpassFilterReq:
	selectors:
	- dataset.DatasetType == "derivative"
	- '["alff", "falff"].includes(suffix)'
	fields:
	BandpassFilter: required
	BandpassFilter:
	selectors:
	- dataset.DatasetType == "derivative"
	- '["alff", "falff"].includes(suffix)'
	fields:
	BandpassFilter:
	level: required
	level_addendum: for suffixes `alff` and `falff`

[effigies]

This should match any of these suffixes.

[effigies]

timeseries.tsv can contain a superset of what's in these other files.

Suggested change

	---
	---
	GeneralTimeSeries:
	selectors:
	- dataset.DatasetType == "derivative"
	- suffix == 'timeseries'
	columns:
	- trans_x: optional
	- trans_y: optional
	- trans_z: optional
	- rot_x: optional
	- rot_y: optional
	- rot_z: optional
	- framewise_displacement: optional
	- rmsd: optional
	- rms: optional
	- non_steady_state_x: optional
	- dvars: optional
	- std_dvars: optional
	- cosine_x: optional
	- legendre: optional
	- compcor: optional
	- melodic_x: optional
	additional_columns: allowed

[yarikoptic]

Just FTR (attn @bendichter) something similarish came up in BEP032 context where for different shape of electrodes we would like to have columns describring the shapes, e.g.

name   shape   shape_radius  shape_width
1      square       n/a         1
2      circle        1         n/a