Contact: [email protected]
Document Status: Approved
Version: 3.0.0
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED" "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14, RFC2119, and RFC8174 when, and only when, they appear in all capitals, as shown here.
cellxgene aims to support the publication, sharing, and exploration of single-cell datasets. Building on those published datasets, cellxgene seeks to create references of the phenotypes and composition of cells that make up human tissues.
Creating references from multiple datasets requires some harmonization of metadata and features in the cellxgene Data Portal. But if that harmonization is too onerous, it will burden the goal of rapid data sharing. cellxgene balances publishing and reference creation needs by requiring datasets hosted in the cellxgene Data Portal to include a small set of metadata readily available from data submitters.
This document describes the schema, a type of contract, that cellxgene requires all datasets to adhere to so that it can enable searching, filtering, and integration of datasets it hosts.
Note that the requirements in the schema are just the minimum required information. Datasets often have additional metadata, which is preserved in datasets submitted to the cellxgene Data Portal.
This schema supports multiple assay types. Each assay takes the form of one or more two-dimensional matrices whose values are quantitative measures of the phenotypes of cells.
The schema additionally describes how the dataset, genes, and cells are annotated to describe the biological and technical characteristics of the data.
This document is organized by:
- General requirements
X
(Matrix layers), which describe the data required for different assaysobs
(Cell metadata), which describe each cell in the datasetvar
andraw.var
(Gene metadata), which describe each gene in the datasetobsm
(Embeddings), which describe each embedding in the datasetuns
(Dataset metadata), which describe the dataset as a whole
AnnData. The canonical data format for the cellxgene Data Portal is HDF5-backed AnnData as written by version 0.8 of the anndata library. Part of the rationale for selecting this format is to allow cellxgene to access both the data and metadata within a single file. The schema requirements and definitions for the AnnData X
, obs
, var
, raw.var
, obsm
, and uns
attributes are described below.
All data submitted to the cellxgene Data Portal is automatically converted to a Seurat V4 object that can be loaded by the R package Seurat. See the Seurat encoding for further information.
Organisms. Data MUST be from a Metazoan organism or SARS-COV-2 and defined in the NCBI organismal classification. For data that is neither Human, Mouse, nor SARS-COV-2, features MUST be translated into orthologous genes from the pinned Human and Mouse gene annotations.
Reserved Names. The names of metadata keys MUST NOT start with "__"
. The names of the metadata keys specified by the schema are reserved and MUST be unique. For example, duplicate "feature_biotype"
keys in AnnData var
are not allowed.
Reserved Names from previous schema versions that have since been deprecated MUST NOT be present in datasets:
Reserved Name | AnnData | Deprecated in |
---|---|---|
ethnicity | obs | 3.0.0 |
ethnicity_ontology_term_id | obs | 3.0.0 |
X_normalization | uns | 3.0.0 |
default_field | uns | 2.0.0 |
layer_descriptions | uns | 2.0.0 |
tags | uns | 2.0.0 |
version | uns | 2.0.0 |
contributors | uns | 1.1.0 |
preprint_doi | uns | 1.1.0 |
project_description | uns | 1.1.0 |
project_links | uns | 1.1.0 |
project_name | uns | 1.1.0 |
publication_doi | uns | 1.1.0 |
Redundant Metadata. It is STRONGLY RECOMMENDED to avoid multiple metadata fields containing identical or similar information.
No Personal Identifiable Information (PII). This is not strictly enforced by validation because it is difficult for software to predict what is and is not PII; however, curators MUST agree to the data submission policies of the cellxgene Data Portal on behalf of data submitters which includes this requirement:
It is my responsibility to ensure that this data is not identifiable. In particular, I commit that I will remove any direct personal identifiers in the metadata portions of the data, and that CZI may further contact me if it believes more work is needed to de-identify it.
This includes names, emails, or other PII for researchers or curators involved in the data generation and submission.
The types below are python3 types. Note that a python3 str
is a sequence of Unicode code points, which is stored null-terminated and UTF-8-encoded by anndata.
The data stored in the X
data matrix is the data that is viewable in cellxgene Explorer. cellxgene does not impose any additional constraints on the X
data matrix.
In any layer, if a matrix has 50% or more values that are zeros, it is STRONGLY RECOMMENDED that the matrix be encoded as a scipy.sparse.csr_matrix
.
cellxgene's matrix layer requirements are tailored to optimize data reuse. Because each assay has different characteristics, the requirements differ by assay type. In general, cellxgene requires submission of "raw" data suitable for computational reuse when a standard raw matrix format exists for an assay. It is STRONGLY RECOMMENDED to also include a "normalized" matrix with processed values ready for data analysis and suitable for visualization in cellxgene Explorer. So that cellxgene's data can be provided in download formats suitable for both R and Python, the schema imposes the following requirements:
- All matrix layers MUST have the same shape, and have the same cell labels and gene labels.
- Because it is impractical to retain all barcodes in raw and normalized matrices, any cell filtering MUST be applied to both. By contrast, those wishing to reuse datasets require access to raw gene expression values, so genes SHOULD NOT be filtered from either dataset. Summarizing, any cell barcodes that are removed from the data MUST be filtered from both raw and normalized matrices and genes SHOULD NOT be filtered from the raw matrix.
- Any genes that publishers wish to filter from the normalized matrix MAY have their values replaced by zeros and MUST be flagged in the column
feature_is_filtered
ofvar
, which will mask them from exploration. - Additional layers provided at author discretion MAY be stored using author-selected keys, but MUST have the same cells and genes as other layers. It is STRONGLY RECOMMENDED that these layers have names that accurately summarize what the numbers in the layer represent (e.g.
"counts_per_million"
,"SCTransform_normalized"
, or"RNA_velocity_unspliced"
).
The following table describes the matrix data and layers requirements that are assay-specific. If an entry in the table is empty, the schema does not have any other requirements on data in those layers beyond the ones listed above.
Assay | "raw" required? | "raw" location | "normalized" required? | "normalized" location |
---|---|---|---|---|
scRNA-seq (UMI, e.g. 10x v3) | REQUIRED. Values MUST be de-duplicated molecule counts. | AnnData.raw.X unless no "normalized" is provided, then AnnData.X |
STRONGLY RECOMMENDED | AnnData.X |
scRNA-seq (non-UMI, e.g. SS2) | REQUIRED. Values MUST be one of read counts (e.g. FeatureCounts) or estimated fragments (e.g. output of RSEM). | AnnData.raw.X unless no "normalized" is provided, then AnnData.X |
STRONGLY RECOMMENDED | AnnData.X |
Accessibility (e.g. ATAC-seq, mC-seq) | NOT REQUIRED | REQUIRED | AnnData.X |
|
cellxgene requires ontology terms to enable search, comparison, and integration of data. Ontology terms for cell metadata MUST use OBO-format identifiers, meaning a CURIE (prefixed identifier) of the form Ontology:Identifier. For example, EFO:0000001 is a term in the Experimental Factor Ontology (EFO).
The most accurate ontology term MUST always be used. If an exact or approximate ontology term is not available, a new term may be requested:
-
For the Cell Ontology, data submitters may suggest a new term and notify the curation team of the pending term request, so that the datasets can be updated once the term is available.
To meet cellxgene schema requirements, the most accurate available CL term MUST be used until the new term is available. For example if
cell_type_ontology_term_id
describes a relay interneuron, but the most accurate available term in the CL ontology is CL:0000099 for interneuron, then the interneuron term can be used to fulfill this requirement and ensures that users searching for "neuron" are able to find these data. If no appropriate high-level term can be found or the cell type is unknown, then the most accurate term is CL:0000003 for native cell. Users will still be able to access more specific cell type annotations that have been submitted with the dataset (but aren't required by the schema). -
For all other ontologies, data submitters may submit a request to the curation team during the submission process.
Terms documented as obsolete in an ontology MUST NOT be used. For example, EFO:0009310 for obsolete_10x v2 was marked as obsolete in EFO version 3.31.0 and replaced by EFO:0009899 for 10x 3' v2.
The following ontology dependencies are pinned for this version of the schema.
Ontology | OBO Prefix | Release | Download |
---|---|---|---|
Cell Ontology | CL | 2022-09-15 | cl.owl |
Experimental Factor Ontology | EFO | 2022-09-15 EFO 3.46.0 | efo.owl |
Human Ancestry Ontology | HANCESTRO | 2022-07-18 (2.6) | hancestro.owl |
Human Developmental Stages | HsapDv | 2020-03-10 | hsapdv.owl |
Mondo Disease Ontology | MONDO | 2022-09-06 | mondo.owl |
Mouse Developmental Stages | MmusDv | 2020-03-10 | mmusdv.owl |
NCBI organismal classification | NCBITaxon | 2022-06-28 | ncbitaxon.owl |
Phenotype And Trait Ontology | PATO | 2022-08-10 | pato.owl |
Uberon multi-species anatomy ontology | UBERON | 2022-08-19 | uberon.owl |
cellxgene requires ENSEMBL identifiers for genes and External RNA Controls Consortium (ERCC) identifiers for RNA Spike-In Control Mixes to ensure that all datasets it stores measure the same features and can therefore be integrated.
The following gene annotation dependencies are pinned for this version of the schema. For multi-organism experiments, cells from any Metazoan organism are allowed as long as orthologs from the following organism annotations are used.
Source | Required version | Download |
---|---|---|
GENCODE (Human) | Human reference GRCh38 (GENCODE v38/Ensembl 104) | gencode.v38.primary_assembly.annotation.gtf |
GENCODE (Mouse) | Mouse reference GRCm39 (GENCODE vM27/Ensembl 104) | gencode.vM27.primary_assembly.annotation.gtf |
ENSEMBL (COVID-19) | SARS-CoV-2 reference (ENSEMBL assembly: ASM985889v3) | Sars_cov_2.ASM985889v3.101.gtf |
ThermoFisher ERCC Spike-Ins | ThermoFisher ERCC RNA Spike-In Control Mixes (Cat # 4456740, 4456739) | cms_095047.txt |
obs
is a pandas.DataFrame
.
Curators MUST annotate the following columns in the obs
dataframe:
Key | assay_ontology_term_id | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Annotator | Curator | ||||||||||||||
Value | categorical with str categories. This MUST be an EFO term and either:
"EFO:0008995" for 10x technology or preferably its most accurate child. An assay based on SMART (Switching Mechanism at the 5' end of the RNA Template) or SMARTer technology SHOULD either be "EFO:0010184" for Smart-like or preferably its most accurate child.Recommended values for specific assays:
|
Key | cell_type_ontology_term_id |
---|---|
Annotator | Curator |
Value | categorical with str categories. This MUST be a CL term.
|
Key | development_stage_ontology_term_id | ||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Annotator | Curator | ||||||||||||||||
Value | categorical with str categories. If unavailable, this MUST be "unknown". If organism_ontolology_term_id is "NCBITaxon:9606" for Homo sapiens, this MUST be the mostaccurate HsapDv term with the following STRONGLY RECOMMENDED:
If organism_ontolology_term_id is "NCBITaxon:10090" for Mus musculus, this MUST be the mostaccurate MmusDv term with the following STRONGLY RECOMMENDED:
Otherwise, for all other organisms this MUST be the most accurate child of UBERON:0000105 for life cycle stage, excluding UBERON:0000071 for death stage. |
Key | disease_ontology_term_id |
---|---|
Annotator | Curator |
Value | categorical with str categories. This MUST be a MONDO term or "PATO:0000461" for normal or healthy.
|
Key | donor_id |
---|---|
Annotator | Curator |
Value | categorical with str categories. This MUST be free-text that identifies a unique individual that data were derived from. It is STRONGLY RECOMMENDED that this identifier be designed so that it is unique to:
It is STRONGLY RECOMMENDED that "pooled" be used for observations from a sample of multiple individuals that were not confidently assigned to a single individual through demultiplexing.It is STRONGLY RECOMMENDED that "unknown" ONLY be used for observations in a dataset when it is not known which observations are from the same individual. |
Key | is_primary_data |
---|---|
Annotator | Curator |
Value | bool . This MUST be True if this is the canonical instance of this cellular observation and False if not. This is commonly False for meta-analyses reusing data or for secondary views of data.
|
Key | organism_ontology_term_id |
---|---|
Annotator | Curator |
Value | categorical with str categories. This MUST be a child of NCBITaxon:33208 for Metazoa.
|
Key | self_reported_ethnicity_ontology_term_id |
---|---|
Annotator | Curator |
Value | categorical with str categories. If organism_ontolology_term_id is "NCBITaxon:9606" for Homo sapiens, this MUST be either a HANCESTRO term, "multiethnic" if more than one ethnicity is reported, or "unknown" if unavailable. Otherwise, for all other organisms this MUST be "na" .
|
Key | sex_ontology_term_id |
---|---|
Annotator | Curator |
Value | categorical with str categories. This MUST be a child of PATO:0001894 for phenotypic sex or "unknown" if unavailable.
|
Key | suspension_type | ||||||||||||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Annotator | Curator | ||||||||||||||||||||||||||||||||||||
Value | categorical with str categories. This MUST be "cell" , "nucleus" , or "na" .This MUST be the correct type for the corresponding assay:
If the assay does not appear in this table, the most appropriate value MUST be selected and the curation team informed during submission so that the assay can be added to the table. |
Key | tissue_ontology_term_id | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Annotator | Curator | ||||||||||
Value | categorical with str categories. This MUST be the UBERON or CL term that best describes the tissue that this cell was derived from, depending on the type of biological sample:
|
When a dataset is uploaded, the cellxgene Data Portal MUST automatically add the matching human-readable name for the corresponding ontology term to the obs
dataframe. Curators MUST NOT annotate the following columns.
Key | assay |
---|---|
Annotator | Data Portal |
Value | categorical with str categories. This MUST be the human-readable name assigned to the value of assay_ontology_term_id .
|
Key | cell_type |
---|---|
Annotator | Data Portal |
Value | categorical with str categories. This MUST be the human-readable name assigned to the value of cell_type_ontology_term_id .
|
Key | development_stage |
---|---|
Annotator | Data Portal |
Value | categorical with str categories. This MUST be "unknown" if set in development_stage_ontology_term_id ; otherwise, this MUST be the human-readable name assigned to the value of development_stage_ontology_term_id .
|
Key | disease |
---|---|
Annotator | Data Portal |
Value | categorical with str categories. This MUST be the human-readable name assigned to the value of disease_ontology_term_id .
|
Key | organism |
---|---|
Annotator | Data Portal |
Value | categorical with str categories. This MUST be the human-readable name assigned to the value of organism_ontology_term_id .
|
Key | self_reported_ethnicity |
---|---|
Annotator | Data Portal |
Value | categorical with str categories. This MUST be "na" , "multiethnic" , or "unknown" if set in self_reported_ethnicity_ontology_term_id ; otherwise, this MUST be the human-readable name assigned to the value of self_reported_ethnicity_ontology_term_id .
|
Key | sex |
---|---|
Annotator | Data Portal |
Value | categorical with str categories. This MUST be "unknown" if set in sex_ontology_term_id ; otherwise, this MUST be the human-readable name assigned to the value of sex_ontology_term_id .
|
Key | tissue |
---|---|
Annotator | Data Portal |
Value | categorical with str categories. This MUST be the human-readable name assigned to the value of tissue_ontology_term_id . " (cell culture)" or " (organoid)" MUST be appended if present in tissue_ontology_term_id .For example, if the tissue_ontology_term_id was curated as "CL:0000057 (cell culture)" , then the value would be "fibroblast (cell culture)" .
|
var
and raw.var
are both of type pandas.DataFrame
.
Curators MUST annotate the following columns in the var
dataframe and if present, the raw.var
dataframe.
Key | index of pandas.DataFrame |
---|---|
Annotator | Curator |
Value | str . If the feature is a gene then this MUST be an ENSEMBL term. If the feature is a RNA Spike-In Control Mix then this MUST be an ERCC Spike-In identifier (e.g. "ERCC-0003" ).The index of the pandas.DataFrame MUST contain unique identifiers for features. If present, the index of raw.var MUST be identical to the index of var . |
Curators MUST annotate the following column only in the var
dataframe. This column MUST NOT be present in raw.var
:
Key | feature_is_filtered |
---|---|
Annotator | Curator |
Value | bool . This MUST be True if the feature was filtered out in the normalized matrix (X ) but is present in the raw matrix (raw.X ). The value for all cells of the given feature in the normalized matrix MUST be 0 . Otherwise, this MUST be False . |
When a dataset is uploaded, cellxgene Data Portal MUST automatically add the matching human-readable name for the corresponding feature biotype, identifier, and the NCBITaxon term for the reference organism to the var
and raw.var
dataframes. Curators MUST NOT annotate the following columns:
Key | feature_biotype |
---|---|
Annotator | Data Portal |
Value | This MUST be "gene" or "spike-in" .
|
Key | feature_name |
---|---|
Annotator | Data Portal |
Value | str . If the feature_biotype is "gene" then this MUST be the human-readable ENSEMBL gene name assigned to the feature identifier in var.index . If the feature_biotype is "spike-in" then this MUST be the ERCC Spike-In identifier appended with " (spike-in control)" .
|
Key | feature_reference | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Annotator | Data Portal | ||||||||||
Value | str . This MUST be the reference organism for a feature:
|
For each str
key, obsm
stores a numpy.ndarray
of shape (n_obs, m)
, where n_obs
is the number of rows in X
and m >= 1
.
To display a dataset in cellxgene Explorer, Curators MUST annotate one or more two-dimensional (m >= 2
) embeddings (e.g. tSNE, UMAP, PCA, spatial coordinates) as numpy.ndarrays
in obsm
. The keys for these embedding MUST be prefixed with "X_"
. The text that follows this prefix is presented to users in the Embedding Choice selector in cellxgene Explorer.
To illustrate, the Krasnow Lab Human Lung Cell Atlas, 10X dataset in the A molecular cell atlas of the human lung from single cell RNA sequencing collection defines two embeddings in obsm
:
"X_Compartment_tSNE"
"X_tSNE"
Users can then choose which embedding is visualized in cellxgene Explorer:
See also default_embedding
in uns
.
uns
is a ordered dictionary with a str
key. Curators MUST annotate the following keys and values in uns
:
Key | schema_version |
---|---|
Annotator | Curator |
Value |
This MUST be "3.0.0" .
|
Key | title |
---|---|
Annotator | Curator |
Value |
str . This text describes and differentiates the dataset from other datasets in the same collection. It is displayed on a page in the cellxgene Data Portal that also has the collection name. To illustrate, the first dataset name in the Cells of the adult human heart collection is "All — Cells of the adult human heart".It is STRONGLY RECOMMENDED that each dataset title in a collection is unique and does not depend on other metadata such as a different assay to disambiguate it from other datasets in the collection.
|
Curators MAY also annotate the following optional keys and values in uns
. If the key is present, then its value MUST NOT be empty.
Key | batch_condition |
---|---|
Annotator | Curator |
Value |
list[str] . str values MUST refer to cell metadata keys in obs . Together, these keys define the batches that a normalization or integration algorithm should be aware of. For example if "patient" and "seqBatch" are keys of vectors of cell metadata, either ["patient"] , ["seqBatch"] , or ["patient", "seqBatch"] are valid values.
|
Key | default_embedding |
---|---|
Annotator | Curator |
Value |
str . The value MUST match a key to an embedding in obsm for the embedding to display by default in cellxgene Explorer.
|
Key | X_approximate_distribution |
---|---|
Annotator | Curator |
Value |
str . cellxgene runs a heuristic to detect the approximate distribution of the data in X so that it can accurately calculate statistical properties of the data. This field enables the curator to override this heuristic and specify the data distribution explicitly. The value MUST be "count" (for data whose distributions are best approximated by counting distributions like Poisson, Binomial, or Negative Binomial) or "normal" (for data whose distributions are best approximated by the Gaussian distribution.)
|
schema v3.0.0
- The canonical data format was updated from AnnData 0.7 to 0.8.
- All references to the "final" matrix has been replaced with "normalized" for clarity.
- General Requirements
- Reserved Names from previous schema versions that have since been deprecated MUST NOT be present.
- Updated pinned ontologies to require the most recent version
- obs (Cell metadata)
- Removed guidance in
assay_ontology_term_id
that allowed clarifying text enclosed in parentheses if there was not an exact match for an assay. - Added
donor_id
- Renamed
ethnicity_ontology_term_id
toself_reported_ethnicity_ontology_term_id
. Added"multiethnic"
value. - Renamed
ethnicity
toself_reported_ethnicity
. Added"multiethnic"
value. - Added
suspension_type
- Removed guidance in
- var and raw.var (Gene metadata)
feature_biotype
must be annotated by the Data Portal and not the Curator.
- uns (Dataset metadata)
- Updated
schema_version
- Deprecated
X_normalization
- Updated
schema v2.0.0 substantially remodeled schema v1.1.0:
-
"must", "should", and select other words have a defined, standard meaning.
-
Curators are responsible for annotating ontology and gene identifiers. The cellxgene Data Portal adds the assigned human-readable names for all identifiers.
-
Documented and pinned the required versions of ontologies and gene annotations used in schema validation.
-
General Requirements
-
AnnData is now the canonical data format. The schema outline and descriptions are AnnData-centric.
-
Metazoan multi-organism data is accepted by the cellxgene Data Portal. For data that is neither Human, Mouse, nor SARS-COV-2, features MUST be translated into orthologous genes from the Human and Mouse gene annotations.
-
Policies for reserved names and redundant metadata are documented.
-
#45 Updated reference to new PII content
-
-
X (matrix layers)
- Added guidance for sparse matrices
- Clarified matrix requirements by assay
-
obs (cell metadata)
- Empty ontology fields are no longer permitted.
- Moved organism from uns to obs
- Clarified requirements and added detailed guidance for assays, tissue, and development stages
- Added ontology for mouse development stages
- Added ontology for sex
- Added
is_primary_data
-
var
- Replaced HGNC gene symbols as
var.index
with ENSEMBL or ERCC spike-in identifiers - Added
feature_name
,index
, andfeature_reference
- Added
feature_is_filtered
- Added requirements for
raw.var
which must be identical tovar
- Replaced HGNC gene symbols as
-
uns
- Added
batch_condition
- Added
X_approximate_distribution
- Replaced
layer_descriptions
withX_normalization
- Replaced
version
which includedcorpora_schema_version
andcorpora_encoding_version
withschema_version
- Deprecated
tags
anddefault_field
presentation metadata - Removed
obs_column_colors
- Added