diff --git a/src/common-principles.md b/src/common-principles.md index 1b8c09a407..461842ddfc 100644 --- a/src/common-principles.md +++ b/src/common-principles.md @@ -430,36 +430,54 @@ NIfTI header. ### Tabular files -Tabular data MUST be saved as tab delimited values (`.tsv`) files, that is, CSV -files where commas are replaced by tabs. Tabs MUST be true tab characters and -MUST NOT be a series of space characters. Each TSV file MUST start with a header -line listing the names of all columns (with the exception of -[physiological and other continuous recordings](modality-specific-files/physiological-and-other-continuous-recordings.md) -as well as [motion recording data](modality-specific-files/motion.md)). +Tabular data MUST be saved as plain-text, tab-delimited values (TSV) files +(with [extension `.tsv`](glossary.md#tsv-extensions)), +that is, [CSV files](https://en.wikipedia.org/wiki/Comma-separated_values) where commas are replaced by tab characters. +Tabs MUST be true tab characters and MUST NOT be a series of space characters. +Tabular data such as continuous physiology recordings typically containing +large numbers of rows MAY be saved as +[compressed tabular files (with extension `.tsv.gz`)](#compressed-tabular-files), +which are introduced below. +Plain-text TSV and compressed TSV are not interchangeable, that is, each section +of the specification prescribes which one MUST be used for the data type at +hand. +Each TSV file MUST start with a header line listing the names of all columns +with two exceptions: + +1. [compressed tabular files](#compressed-tabular-files), + for which column names are defined in a sidecar metadata + [JSON object](https://www.json.org/json-en.html) described below; and +1. [motion recording data](modality-specific-files/motion.md), + which use plain-text TSV and columns are defined as described + in its corresponding section of the specifications. + It is RECOMMENDED that the column names in the header of the TSV file are written in [`snake_case`](https://en.wikipedia.org/wiki/Snake_case) with the first letter in lower case (for example, `variable_name`, not `Variable_name`). -As for all other data in the TSV files, column names MUST be separated with tabs. +Column names defined in the header MUST be separated with tabs as for the data contents. Furthermore, column names MUST NOT be blank (that is, an empty string) and MUST NOT be duplicated within a single TSV file. -String values containing tabs MUST be escaped using double -quotes. Missing and non-applicable values MUST be coded as `n/a`. Numerical -values MUST employ the dot (`.`) as decimal separator and MAY be specified +String values containing tabs MUST be escaped using double quotes. +Missing and non-applicable values MUST be coded as `n/a`. +Numerical values MUST employ the dot (`.`) as decimal separator and MAY be specified in scientific notation, using `e` or `E` to separate the significand from the -exponent. TSV files MUST be in UTF-8 encoding. +exponent. +TSV files MUST be in UTF-8 encoding. Example: ```Text -onset duration response_time correct stop_trial go_trial -200 200 0 n/a n/a n/a +onset duration response_time trial_type trial_extra +200 20.0 15.8 word 中国人 +240 5.0 17.34e-1 visual n/a ``` -**Note**: The TSV examples in this document (like the one above this note) -are occasionally formatted using space characters instead of tabs to improve -human readability. -Directly copying and then pasting these examples from the specification -for use in new BIDS datasets can lead to errors and is discouraged. +!!! warning "Attention" + + The TSV examples in this document (like the one above this note) are occasionally + formatted using space characters instead of tabs to improve human readability. + Directly copying and then pasting these examples from the specification + for use in new BIDS datasets can lead to errors and is discouraged. Tabular files MAY be optionally accompanied by a simple data dictionary in the form of a JSON [object](https://www.json.org/json-en.html) @@ -536,12 +554,38 @@ like in the example below. "F": { "Description": "Female", "TermURL": "https://www.ncbi.nlm.nih.gov/mesh/68005260" - }, + } } } } ``` +### Compressed tabular files + +Large tabular information, such as physiological recordings, MUST be stored with +[compressed tab-delineated (TSV.GZ) files](glossary.md#tsvgz-extensions) when +so established by the specifications. +Rules for formatting plain-text tabular files apply to TSVGZ files with three exceptions: + +1. The contents of TSVGZ files MUST be compressed with + [gzip](https://datatracker.ietf.org/doc/html/rfc1952). +1. Compressed tabular files MUST NOT contain a header in the first row + indicating the column names. +1. TSVGZ files MUST have an associated JSON file that defines the columns in the tabular file. + +!!! warning "Attention" + + In contrast to plain-text TSV files, + compressed tabular files files MUST NOT include a header line. + Column names MUST be provided in the JSON file with the + [`Columns`](glossary.md#columns-metadata) field. + Each column MAY additionally be described with a column description, + as described in [Tabular files](#tabular-files). + + TSVGZ are header-less to improve compatibility with existing software + (for example, FSL, or PNM), and to facilitate the support for other file formats + in the future. + ### Key-value files (dictionaries) JavaScript Object Notation (JSON) files MUST be used for storing key-value diff --git a/src/modality-specific-files/physiological-and-other-continuous-recordings.md b/src/modality-specific-files/physiological-and-other-continuous-recordings.md index 38f9842ae3..1cde990e41 100644 --- a/src/modality-specific-files/physiological-and-other-continuous-recordings.md +++ b/src/modality-specific-files/physiological-and-other-continuous-recordings.md @@ -2,12 +2,9 @@ Physiological recordings such as cardiac and respiratory signals and other continuous measures (such as parameters of a film or audio stimuli) MAY be -specified using two files: - -1. a [gzip](https://datatracker.ietf.org/doc/html/rfc1952) - compressed TSV file with data (without header line) - -1. a JSON file for storing metadata fields (see below) +specified using a [compressed tabular file](../common-principles.md#compressed-tabular-files) +([TSVGZ file](../glossary.md#tsvgz-extensions)) and a corresponding +JSON file for storing metadata fields (see below). !!! example "Example datasets" @@ -38,8 +35,12 @@ before the suffix. For example for the file `sub-control01_task-nback_run-1_bold.nii.gz`, `` would correspond to `sub-control01_task-nback_run-1`. -Note that when supplying a `*_.tsv.gz` file, an accompanying -`*_.json` MUST be supplied as well. +!!! note "TSVGZ headers are specified in metadata files." + + TSVGZ files MUST NOT include a header line, + as established by the [common-principles](../common-principles.md#compressed-tabular-files). + As a result, when supplying a `*_.tsv.gz` file, an accompanying + `*_.json` MUST be supplied as well. The [`recording-