docs: added notes about validation

biosimulators · Oct 8, 2021 · be27b8d · be27b8d
1 parent 02ab938
commit be27b8d
Show file tree

Hide file tree

Showing 4 changed files with 88 additions and 50 deletions.
diff --git a/.... Validating models and simulations/1. Validating models (e.g., CellML, SBML files).ipynb b/.... Validating models and simulations/1. Validating models (e.g., CellML, SBML files).ipynb
@@ -26,7 +26,7 @@
     "* [XPP ODE format](http://www.math.pitt.edu/~bard/xpp/help/xppodes.html)\n",
     "\n",
     "<div class=\"alert alert-block alert-info\">\n",
-    "    BioSimulators integrates community-contributed validators for each model language. For some model languages, these validators provide limited validation and/or limited reports of errors. Please contact the <a href=\"mailto:[email protected]\">BioSimulators Team</a> to contribute additional validators.\n",
+    "    BioSimulators integrates community-contributed validators for each model language. For some model languages, these validators provide limited validation and/or limited reports of errors. We welcome contributions of improved validation tools.\n",
     "</div>"
    ]
   },
@@ -799,7 +799,7 @@
      "name": "stderr",
      "output_type": "stream",
      "text": [
-      "/tmp/ipykernel_277/783546003.py:5: UserWarning: - The value of the 'sboTerm' attribute on a <species> is expected to be an SBO identifier (http://www.biomodels.net/SBO/). In SBML Level 2 prior to Version 4 it is expected to refer to a participant physical type (i.e., terms derived from SBO:0000236, \"participant physical type\"); in Versions 4 and above it is expected to refer to a material entity (i.e., terms derived from SBO:0000240, \"material entity\").\n",
+      "/tmp/ipykernel_77/783546003.py:5: UserWarning: - The value of the 'sboTerm' attribute on a <species> is expected to be an SBO identifier (http://www.biomodels.net/SBO/). In SBML Level 2 prior to Version 4 it is expected to refer to a participant physical type (i.e., terms derived from SBO:0000236, \"participant physical type\"); in Versions 4 and above it is expected to refer to a material entity (i.e., terms derived from SBO:0000240, \"material entity\").\n",
       "  Reference: L2V4 Section 5\n",
       "   SBO term 'SBO:0000014' on the <species> is not in the appropriate branch.\n",
       "  \n",

diff --git a/...lidating models and simulations/2. Validating simulation experiments (SED-ML files).ipynb b/...lidating models and simulations/2. Validating simulation experiments (SED-ML files).ipynb
@@ -16,7 +16,7 @@
     "This tutorial illustrates how to validate simulation experiments encoded in the [Simulation Experiment Description Markup Language (SED-ML)](http://sed-ml.org/). This format uses [URNs](https://sed-ml.org/urns.html) to describe the language of each model. This format uses the [Kinetic Simulation Algorithm Ontology (KiSAO)](https://github.com/SED-ML/KiSAO) to describe algorithms and algorithm parameters.\n",
     "\n",
     "<div class=\"alert alert-block alert-info\">\n",
-    "    The methods described here do not validate the sources of the models (e.g., CellML, SBML files) of SED-ML files or targets for changes to models or for outputs of simulations of models. More comprehensive validation is available through validation of entire simulation projects (COMBINE/OMEX archives). This encompasses validation of the sources of models in SED-ML files and targets for model changes and simulation observables. Please see the <a href=\"4.%20Validating%20simulation%20projects%20(COMBINE-OMEX%20archives).ipynb\">simulation project validation tutorial</a> for more information.\n",
+    "    The methods described here do not validate the sources of the models (e.g., CellML, SBML files) of SED-ML files or targets for changes to models or for outputs of simulations of models. More comprehensive validation is available through validation of entire simulation projects (COMBINE/OMEX archives). This encompasses validation of the sources of models in SED-ML files and targets for models for model changes and simulation observables. Please see the <a href=\"4.%20Validating%20simulation%20projects%20(COMBINE-OMEX%20archives).ipynb\">simulation project validation tutorial</a> for more information.\n",
     "</div>\n",
     "\n",
     "<div class=\"alert alert-block alert-info\">\n",
@@ -287,35 +287,35 @@
       "text/plain": [
        "{'level': 1,\n",
        " 'version': 3,\n",
-       " 'models': [<biosimulators_utils.sedml.data_model.Model at 0x7fd97502f220>],\n",
-       " 'simulations': [<biosimulators_utils.sedml.data_model.UniformTimeCourseSimulation at 0x7fd97502f9a0>],\n",
-       " 'tasks': [<biosimulators_utils.sedml.data_model.Task at 0x7fd97502f700>],\n",
-       " 'data_generators': [<biosimulators_utils.sedml.data_model.DataGenerator at 0x7fd97502f2e0>,\n",
-       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7fd97502f040>,\n",
-       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7fd97502ff10>,\n",
-       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7fd975036670>,\n",
-       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7fd975036af0>,\n",
-       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7fd975036c10>,\n",
-       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7fd975036ca0>,\n",
-       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7fd975036d90>,\n",
-       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7fd975036e50>,\n",
-       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7fd975036f40>,\n",
-       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7fd975036280>,\n",
-       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7fd975036460>,\n",
-       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7fd975036580>,\n",
-       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7fd975036f70>,\n",
-       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7fd9750364c0>,\n",
-       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7fd975036520>,\n",
-       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7fd97506dc70>,\n",
-       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7fd97506da30>,\n",
-       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7fd97506dcd0>,\n",
-       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7fd97506db50>,\n",
-       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7fd97506d7f0>],\n",
-       " 'outputs': [<biosimulators_utils.sedml.data_model.Report at 0x7fd97502f8b0>,\n",
-       "  <biosimulators_utils.sedml.data_model.Plot2D at 0x7fd97506dfa0>,\n",
-       "  <biosimulators_utils.sedml.data_model.Plot2D at 0x7fd97500ddc0>,\n",
-       "  <biosimulators_utils.sedml.data_model.Plot2D at 0x7fd9748e3280>,\n",
-       "  <biosimulators_utils.sedml.data_model.Plot2D at 0x7fd975039be0>],\n",
+       " 'models': [<biosimulators_utils.sedml.data_model.Model at 0x7f9db42d1a60>],\n",
+       " 'simulations': [<biosimulators_utils.sedml.data_model.UniformTimeCourseSimulation at 0x7f9db42d1d90>],\n",
+       " 'tasks': [<biosimulators_utils.sedml.data_model.Task at 0x7f9db42d1940>],\n",
+       " 'data_generators': [<biosimulators_utils.sedml.data_model.DataGenerator at 0x7f9db42d1700>,\n",
+       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7f9df8173880>,\n",
+       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7f9db431b0d0>,\n",
+       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7f9db42d1b50>,\n",
+       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7f9db431b370>,\n",
+       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7f9db42b6340>,\n",
+       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7f9db42b6250>,\n",
+       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7f9db42b6bb0>,\n",
+       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7f9db42b6ca0>,\n",
+       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7f9db42b6520>,\n",
+       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7f9db42b6d90>,\n",
+       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7f9db42b6610>,\n",
+       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7f9db42b6ac0>,\n",
+       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7f9db42b6f70>,\n",
+       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7f9db42b68b0>,\n",
+       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7f9db42b69a0>,\n",
+       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7f9db42b69d0>,\n",
+       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7f9db42b6700>,\n",
+       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7f9db42b6e80>,\n",
+       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7f9db42b60d0>,\n",
+       "  <biosimulators_utils.sedml.data_model.DataGenerator at 0x7f9db42b6dc0>],\n",
+       " 'outputs': [<biosimulators_utils.sedml.data_model.Report at 0x7f9db42d1610>,\n",
+       "  <biosimulators_utils.sedml.data_model.Plot2D at 0x7f9db42454c0>,\n",
+       "  <biosimulators_utils.sedml.data_model.Plot2D at 0x7f9db42ae520>,\n",
+       "  <biosimulators_utils.sedml.data_model.Plot2D at 0x7f9db3b82fd0>,\n",
+       "  <biosimulators_utils.sedml.data_model.Plot2D at 0x7f9db42db0a0>],\n",
        " 'metadata': None}"
       ]
      },
@@ -337,9 +337,9 @@
     {
      "data": {
       "text/plain": [
-       "{'__init__': <bound method SedDocument.__init__ of <biosimulators_utils.sedml.data_model.SedDocument object at 0x7fd97502fd30>>,\n",
-       " 'is_equal': <bound method SedDocument.is_equal of <biosimulators_utils.sedml.data_model.SedDocument object at 0x7fd97502fd30>>,\n",
-       " 'to_tuple': <bound method SedDocument.to_tuple of <biosimulators_utils.sedml.data_model.SedDocument object at 0x7fd97502fd30>>}"
+       "{'__init__': <bound method SedDocument.__init__ of <biosimulators_utils.sedml.data_model.SedDocument object at 0x7f9db42d1eb0>>,\n",
+       " 'is_equal': <bound method SedDocument.is_equal of <biosimulators_utils.sedml.data_model.SedDocument object at 0x7f9db42d1eb0>>,\n",
+       " 'to_tuple': <bound method SedDocument.to_tuple of <biosimulators_utils.sedml.data_model.SedDocument object at 0x7f9db42d1eb0>>}"
       ]
      },
      "execution_count": 9,

diff --git a/...s/3. Validating models and simulations/3. Validating metadata (OMEX Metadata files).ipynb b/...s/3. Validating models and simulations/3. Validating metadata (OMEX Metadata files).ipynb
@@ -15,9 +15,17 @@
    "source": [
     "This tutorial illustrates how to validate [OMEX Metadata](https://sys-bio.github.io/libOmexMeta/) files. OMEX Metadata is a format for capturing metadata about COMBINE/OMEX archives and/or individual items inside archives, such as the organism represented by a model, the author of a simulation experiment, or the date when an archive was created. OMEX Metadata represents metadata as sets of [Resource Description Framework (RDF)](https://en.wikipedia.org/wiki/Resource_Description_Framework) [triples](https://en.wikipedia.org/wiki/Semantic_triple) of subjects (e.g., model in an archive), predicates (e.g., organism that the model represents), and objects (e.g., *Escherichia coli*) that represent entities and relationships between them. OMEX Metadata can flexibly capture a broad range of metadata in machine-readable form.\n",
     "\n",
-    "BioSimulators supports two schemas for OMEX Metadata files:\n",
-    "* **RDF** (`rdf_triples`): Supports anything that can be encoded in RDF. Validation is performed using [libOmexMeta](https://sys-bio.github.io/libOmexMeta/).\n",
-    "* **BioSimulations** (`biosimulations`): We recommend using the BioSimulations schema. This schema imposes additional requirements beyond the RDF schema for minimal metdata about a COMBINE archive, including a title, at least one creator, and a license. This schema is required to submit a simulation project to the BioSimulations repository. More information about this schema is available at [https://biosimulators.org](https://biosimulators.org/conventions/metadata)."
+    "BioSimulators supports two minimum information conventions for OMEX Metadata files:\n",
+    "* **OMEX Metadata** (`rdf_triples`): No minimum set of information is required. This convention supports anything that can be encoded in RDF. Validation is performed using [libOmexMeta](https://sys-bio.github.io/libOmexMeta/).\n",
+    "* **BioSimulations** (`biosimulations`): We recommend using the BioSimulations minimum information convention. This convention imposes additional requirements beyond the OEMX Metadata convention for minimal metdata about a COMBINE archive, including a title, at least one creator, and a license. More information about this convention is available at [https://biosimulators.org](https://biosimulators.org/conventions/metadata).\n",
+    "\n",
+    "<div class=\"alert alert-block alert-info\">\n",
+    "    OMEX Metadata files can describe thumbnails for simulation projects. OMEX Metadata files can be validated independently from these thumbnails. However, this will not validate the thumbnails. More comprehensive validation, including related images, is available through the validation of simulation projects (COMBINE/OMEX archives). Please see the <a href=\"4.%20Validating%20simulation%20projects%20(COMBINE-OMEX%20archives).ipynb\">simulation project validation tutorial</a> for more information.\n",
+    "</div>\n",
+    "\n",
+    "<div class=\"alert alert-block alert-info\">\n",
+    "    BioSimulations' minimum metadata conventions are required for publication to BioSimulations'. Currently, publication to BioSimulations is also restricted to RDF-XML OMEX Metadata files.\n",
+    "</div>"
    ]
   },
   {

diff --git a/...ng models and simulations/4. Validating simulation projects (COMBINE-OMEX archives).ipynb b/...ng models and simulations/4. Validating simulation projects (COMBINE-OMEX archives).ipynb
@@ -27,6 +27,10 @@
     "\n",
     "<div class=\"alert alert-block alert-info\">\n",
     "    SED-ML is under active development. The SED-ML community is working toward a single interpretation of SED-ML. BioSimulators supports the interpretation of SED-ML described at <a href=\"https://biosimulators.org/conventions/simulation-experiments\">https://biosimulators.org</a>.\n",
+    "</div>\n",
+    "\n",
+    "<div class=\"alert alert-block alert-info\">\n",
+    "    BioSimulations relies on community-contributed validation tools for each model language to validate models. The thoroughness of these tools varies. In addition, some tools currently provide limited diagnostic information. We welcome contributions of improved validation tools.\n",
     "</div>"
    ]
   },
@@ -1775,7 +1779,7 @@
      "name": "stderr",
      "output_type": "stream",
      "text": [
-      "/tmp/ipykernel_579/783546003.py:5: UserWarning: - The SED-ML file at location `./simulation_1.sedml` may be invalid.\n",
+      "/tmp/ipykernel_307/783546003.py:5: UserWarning: - The SED-ML file at location `./simulation_1.sedml` may be invalid.\n",
       "  - Model `Ciliberto2003_Morphogenesis` may be invalid.\n",
       "    - The model file `BIOMD0000000297_url.xml` may be invalid.\n",
       "      - The value of the 'sboTerm' attribute on a <species> is expected to be an SBO identifier (http://www.biomodels.net/SBO/). In SBML Level 2 prior to Version 4 it is expected to refer to a participant physical type (i.e., terms derived from SBO:0000236, \"participant physical type\"); in Versions 4 and above it is expected to refer to a material entity (i.e., terms derived from SBO:0000240, \"material entity\").\n",
@@ -2304,14 +2308,14 @@
     {
      "data": {
       "text/plain": [
-       "{'contents': [<biosimulators_utils.combine.data_model.CombineArchiveContent at 0x7f2f320a1220>,\n",
-       "  <biosimulators_utils.combine.data_model.CombineArchiveContent at 0x7f2f320a11c0>,\n",
-       "  <biosimulators_utils.combine.data_model.CombineArchiveContent at 0x7f2f320a12b0>,\n",
-       "  <biosimulators_utils.combine.data_model.CombineArchiveContent at 0x7f2f320a12e0>,\n",
-       "  <biosimulators_utils.combine.data_model.CombineArchiveContent at 0x7f2f320a1310>,\n",
-       "  <biosimulators_utils.combine.data_model.CombineArchiveContent at 0x7f2f320a1340>,\n",
-       "  <biosimulators_utils.combine.data_model.CombineArchiveContent at 0x7f2f320a1370>,\n",
-       "  <biosimulators_utils.combine.data_model.CombineArchiveContent at 0x7f2f320b4ee0>],\n",
+       "{'contents': [<biosimulators_utils.combine.data_model.CombineArchiveContent at 0x7f04cea0f220>,\n",
+       "  <biosimulators_utils.combine.data_model.CombineArchiveContent at 0x7f04cea0f1c0>,\n",
+       "  <biosimulators_utils.combine.data_model.CombineArchiveContent at 0x7f04cea0f2b0>,\n",
+       "  <biosimulators_utils.combine.data_model.CombineArchiveContent at 0x7f04cea0f2e0>,\n",
+       "  <biosimulators_utils.combine.data_model.CombineArchiveContent at 0x7f04cea0f310>,\n",
+       "  <biosimulators_utils.combine.data_model.CombineArchiveContent at 0x7f04cea0f340>,\n",
+       "  <biosimulators_utils.combine.data_model.CombineArchiveContent at 0x7f04cea0f370>,\n",
+       "  <biosimulators_utils.combine.data_model.CombineArchiveContent at 0x7f04cea14ee0>],\n",
        " 'description': None,\n",
        " 'authors': [],\n",
        " 'created': None,\n",
@@ -2381,9 +2385,9 @@
     "| Validate OMEX Metadata files               | `validateOmexMetadata`    | `VALIDATE_OMEX_METADATA`     | `VALIDATE_OMEX_METADATA`       | `True`           |\n",
     "| Validate image files                       | `validateImages`          | `VALIDATE_IMAGES`            | `VALIDATE_IMAGES`              | `True`           |\n",
     "\n",
-    "Two OMEX Metadata schemas are supported:\n",
-    "* **RDF** (`rdf_triples`): Supports anything that can be encoded in RDF. Validation is performed using [libOmexMeta](https://sys-bio.github.io/libOmexMeta/).\n",
-    "* **BioSimulations** (`BioSimulations`): We recommend using the BioSimulations schema. This schema imposes additional requirements beyond the RDF schema for minimal metdata about a COMBINE archive, including a title, at least one creator, and a license. This schema is required to submit a simulation project to the BioSimulations repository. More information about this schema is available at [https://biosimulators.org](https://biosimulators.org/conventions/metadata)."
+    "Two conventions for minimum metadata are supported:\n",
+    "* **OMEX Metadata** (`rdf_triples`): This convention does not require any minimal set of metadata. This convention supports anything that can be encoded in RDF. Validation is performed using [libOmexMeta](https://sys-bio.github.io/libOmexMeta/).\n",
+    "* **BioSimulations** (`BioSimulations`): We recommend using the BioSimulations convention. This convention imposes additional requirements beyond the OMEX Metadata convention for minimal metdata about a simulation project, including a title, at least one creator, and a license. More information about this convention is available at [https://biosimulators.org](https://biosimulators.org/conventions/metadata)."
    ]
   },
   {
@@ -2406,6 +2410,32 @@
     "archive = CombineArchiveReader().run(archive_filename, archive_dirname, config=config)\n",
     "errors, warnings = validate(archive, archive_dirname, config=config)"
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "b681c477-a44a-4a44-a6eb-4cf004df41fa",
+   "metadata": {},
+   "source": [
+    "<div class=\"alert alert-block alert-info\">\n",
+    "    Plain zip archives can be validated by setting <code>VALIDATE_OMEX_MANIFESTS=False</code>.\n",
+    "</div>\n",
+    "\n",
+    "<div class=\"alert alert-block alert-info\">\n",
+    "    Comprehensive validation of SED-ML requires <code>VALIDATE_SEDML=True</code> and <code>VALIDATE_SEDML_MODELS=True</code>. If the latter is false, the sources of the models in SED-ML files and targets to these models will not be validated.\n",
+    "</div>\n",
+    "\n",
+    "<div class=\"alert alert-block alert-info\">\n",
+    "    OMEX Manifests are required for validating OMEX Metadata files. As a result, <code>VALIDATE_OMEX_METADATA=False</code> should often be used together with <code>VALIDATE_OMEX_MANIFESTS=False</code>.\n",
+    "</div>\n",
+    "\n",
+    "<div class=\"alert alert-block alert-info\">\n",
+    "    When OMEX Metadata files describe thumbnail images (e.g., for projects), <code>VALIDATE_IMAGES=True</code> is required to comprehensively validate OMEX Metadata files, including linked images.\n",
+    "</div>\n",
+    "\n",
+    "<div class=\"alert alert-block alert-info\">\n",
+    "    BioSimulations' minimum metadata conventions are required for publication to BioSimulations'. Currently, publication to BioSimulations is also restricted to RDF-XML OMEX Metadata files.\n",
+    "</div>"
+   ]
   }
  ],
  "metadata": {