[ENH] Add compressed TSV files to the common principles

src/common-principles.md

@@ -430,36 +430,54 @@ NIfTI header.
 
 ### Tabular files
 
-Tabular data MUST be saved as tab delimited values (`.tsv`) files, that is, CSV
-files where commas are replaced by tabs. Tabs MUST be true tab characters and
-MUST NOT be a series of space characters. Each TSV file MUST start with a header
-line listing the names of all columns (with the exception of
-[physiological and other continuous recordings](modality-specific-files/physiological-and-other-continuous-recordings.md)
-as well as [motion recording data](modality-specific-files/motion.md)).
+Tabular data MUST be saved as plain-text, tab-delimited values (TSV) files
+(with [extension `.tsv`](glossary.md#tsv-extensions)),
+that is, CSV files where commas are replaced by tab characters.
+Tabs MUST be true tab characters and MUST NOT be a series of space characters.
+Tabular data such as continuous physiology recordings typically containing
+large numbers of rows MAY be saved as
+[compressed tabular files (with extension `.tsv.gz`)](#compressed-tabular-files),
+which are introduced below.
+Plain-text TSV and compressed TSV are not interchangeable, that is, each section
+of the specification prescribes which one SHOULD be used for the data type at
+hand.
+Each TSV file MUST start with a header line listing the names of all columns
+with two exceptions:
+
+1.  [compressed tabular files](#compressed-tabular-files),
+    for which column names are defined in a sidecar metadata
+    [JSON object](https://www.json.org/json-en.html) described below; and
+1.  [motion recording data](modality-specific-files/motion.md),
+    which use plain-text TSV and columns are defined as described
+    in its corresponding section of the specifications.
+
 It is RECOMMENDED that the column names in the header of the TSV file are
 written in [`snake_case`](https://en.wikipedia.org/wiki/Snake_case) with the
 first letter in lower case (for example, `variable_name`, not `Variable_name`).
-As for all other data in the TSV files, column names MUST be separated with tabs.
+Column names defined in the header MUST be separated with tabs as for the data contents.
 Furthermore, column names MUST NOT be blank (that is, an empty string) and MUST NOT
 be duplicated within a single TSV file.
-String values containing tabs MUST be escaped using double
-quotes. Missing and non-applicable values MUST be coded as `n/a`. Numerical
-values MUST employ the dot (`.`) as decimal separator and MAY be specified
+String values containing tabs MUST be escaped using double quotes.
+Missing and non-applicable values MUST be coded as `n/a`.
+Numerical values MUST employ the dot (`.`) as decimal separator and MAY be specified
 in scientific notation, using `e` or `E` to separate the significand from the
-exponent. TSV files MUST be in UTF-8 encoding.
+exponent.
+TSV files MUST be in UTF-8 encoding.
 
 Example:
 
 ```Text
-onset	duration	response_time	correct	stop_trial	go_trial
-200	200	0	n/a	n/a	n/a
+onset   duration    response_time   trial_type        trial_extra
+200     20.0        15.8            word              中国人
+240     5.0         17.34e-1        visual            n/a
 ```
 
-**Note**: The TSV examples in this document (like the one above this note)
-are occasionally formatted using space characters instead of tabs to improve
-human readability.
-Directly copying and then pasting these examples from the specification
-for use in new BIDS datasets can lead to errors and is discouraged.
+!!! warning "Attention"
+
+    The TSV examples in this document (like the one above this note) are occasionally
+    formatted using space characters instead of tabs to improve human readability.
+    Directly copying and then pasting these examples from the specification
+    for use in new BIDS datasets can lead to errors and is discouraged.
 
 Tabular files MAY be optionally accompanied by a simple data dictionary
 in the form of a JSON [object](https://www.json.org/json-en.html)
@@ -536,12 +554,40 @@ like in the example below.
             "F": {
                 "Description": "Female",
                 "TermURL": "https://www.ncbi.nlm.nih.gov/mesh/68005260"
-            },
+            }
         }
     }
 }
 ```
 
+### Compressed tabular files
+
+Large tabular information, such as physiological recordings, SHOULD be stored with
+[compressed tab-delineated (TSVGZ) files](glossary.md#tsvgz-extensions) when
+so established by the specifications.
+Rules for formatting plain-text tabular files apply to TSVGZ files with three exceptions:
+
+1.  The contents of TSVGZ files MUST be compressed with
+    [gzip](https://datatracker.ietf.org/doc/html/rfc1952).
+1.  Compressed tabular files MUST NOT contain a header in the first row
+    indicating the column names.
+1.  TSVGZ files MUST be accompanied by a JSON file with the same name as their
+    corresponding tabular file but with a `.json` extension.
+
+
+!!! warning "Attention"
+
+    In contrast to plain-text TSV files,
+    compressed tabular files files MUST NOT include a header line.
+    Column names MUST be provided in the JSON file with the
+    [`Columns`](glossary.md#columns-metadata) field.
+    Each column MAY additionally be described with a column description,
+    as described in [Tabular files](#tabular-files).
+
+    TSVGZ are header-less to improve compatibility with existing software
+    (for example, FSL, or PNM), and to facilitate the support for other file formats
+    in the future.
+
 ### Key-value files (dictionaries)
 
 JavaScript Object Notation (JSON) files MUST be used for storing key-value

src/modality-specific-files/physiological-and-other-continuous-recordings.md

@@ -2,12 +2,9 @@
 
 Physiological recordings such as cardiac and respiratory signals and other
 continuous measures (such as parameters of a film or audio stimuli) MAY be
-specified using two files:
-
-1.  a [gzip](https://datatracker.ietf.org/doc/html/rfc1952)
-    compressed TSV file with data (without header line)
-
-1.  a JSON file for storing metadata fields (see below)
+specified using a [compressed tabular file](../common-principles.md#compressed-tabular-files)
+([TSVGZ file](../glossary.md#tsvgz-extensions)) and a corresponding
+JSON file for storing metadata fields (see below).
 
 !!! example "Example datasets"
 
@@ -38,8 +35,12 @@ before the suffix.
 For example for the file `sub-control01_task-nback_run-1_bold.nii.gz`,
 `<matches>` would correspond to `sub-control01_task-nback_run-1`.
 
-Note that when supplying a `*_<physio|stim>.tsv.gz` file, an accompanying
-`*_<physio|stim>.json` MUST be supplied as well.
+!!! note "TSVGZ headers are specified in metadata files."
+
+    TSVGZ files MUST NOT include a header line,
+    as established by the [common-principles](../common-principles.md#compressed-tabular-files).
+    As a result, when supplying a `*_<physio|stim>.tsv.gz` file, an accompanying
+    `*_<physio|stim>.json` MUST be supplied as well.
 
 The [`recording-<label>`](../appendices/entities.md#recording)
 entity MAY be used to distinguish between several recording files.
@@ -66,15 +67,6 @@ A guide for using macros can be found at
 Additional metadata may be included as in
 [any TSV file](../common-principles.md#tabular-files) to specify, for
 example, the units of the recorded time series.
-Please note that, in contrast to other TSV files in BIDS, the TSV files specified
-for physiological and other continuous recordings *do not* include a header
-line.
-Instead the name of columns are specified in the JSON file (see `Columns` field).
-This is to improve compatibility with existing software (for example, FSL, PNM)
-as well as to make support for other file formats possible in the future.
-As in any TSV file, column names MUST NOT be blank (that is, an empty string),
-and MUST NOT be duplicated within a single JSON file describing a headerless
-TSV file.
 
 Example `*_physio.tsv.gz`: