Discussion_Limitations

Discussion & Limitations & Results

Systems biology studies generate many data files for the simulation of biological systems. These data files can be stored in COMBINE archives, standardized containers that facilitate the exchange of information from simulation studies [1], such as the fully-featured COMBINE archive described by Scharm & Waltemath [2].

COMBINE archive

As a result of this project, we built a fully-featured COMBINE archive ('blood, sweat and tears') of the Bachmann model [3] using the web platform CombineArchiveWeb. We used a similar folder structure as suggested by Scharm & Waltemath, which is presented in the section 'Structure of COMBINE archive'. We could upload all files from our repository, but report XML files for validations. On this step, the CobineArchiveWeb interface throws the following error: "Unknown Error: Cannot upload file". We have made a new issue in the GitHub repository from WebCAT (issue #146) but did not encounter the settlement of the problem during the course of the project.

After creating the COMBINE archive, we selected the option "Simulate" to automatically run the archive in SED-ML WebTools. This simulation shows the species and its behaviour in the reactions, but only from the default simulation. We have previously tested a simulation without the Default SED-ML file, which functioned too, showing the result of one of our simulations. This point is registered in another issue (#147) in the GitHub of the developer of CombineArchiveWeb. In addition, the COMBINE archive was downloaded and all files were successfully simulated in Tellurium v2.2.0 as described here.

Creation and simulation of SED-ML files

As part of this COMBINE archive we created SED-ML files to reproduce the simulation experiments presented by Bachmann et al. based on published models and information. Initially, we attempted to generate and simulate files in COPASI, a well-established tool for systems biology. While it allows to easily modify model parameters via the user interface, COPASI is limited in its functionality to support simulations using more than one model. This is, however, required in our case, for instance to run simulations in the wildtype and overexpression condition in parallel. We therefore changed our approach and eventually created and simulated SED-ML files using the Python platform Tellurium as described in our documentation.

Reproducibility of experiments

While we were able to re-create some (sub-) figures of the original paper using the model selected for this COMBINE archive, we encountered several problems. These will be addressed in separate GitHub issues for future reference (maybe for the authors of the original paper).

• Some parameters required to simulate specific experimental conditions were not included in the model (e.g. SHP1oe).
• Some model outputs are defined by observation functions and were not included as parameters in the model itself (e.g. pJAK2.au, tSTAT5). It probably would be possible to define these parameters either in the SBML file itself or in the experiment-specific SED-ML files based on information included in the supplementary material. However, we did not address this issue given the limited time frame of this project.
• Several experiments presented in the paper are based on testing a range of Epo levels. We could not fully reproduce these experiments as it was not clear how to simulate different Epo concentrations using the published model.
• The knockout of CIS and/or SOCS3 could not be simulated using the information provided in the paper.

SBGN exchangeability

To foster the subsequent use of standardized graphical representations and increase the added value of the work put in the creation of graphical maps, the exchangeability is one crucial point. Lowering support and delayed or even discontinued adaptation of published tools to actualised standards are troublesome for future projects. Thus, we wanted to provide our SBGN model in a variety of formats to allow recurrence on the substitute files when the primary files are not applicable in a specific future application. Finally, we were able to provide different SBGN-ML 0.2 codes as well as a GraphML version of our model. The CellDesigner export did not work and also we provided no SBGN-ML 0.3 because we were not able to validate this with at least one other tool than Newt editor. Unfortunately, we were not able to further investigate the platform specific problems with exchange formats in detail as we experienced them, but provide a documentation for the community. During the time of our project, the CellDesigner export from Newt editor is already adressed in the discussion to an existing issue on GitHub (https://github.com/iVis-at-Bilkent/newt/issues/498). Also, we created a new issue adressing the manual modifications to the provided SBGN-ML version 0.2 main file to make it exchangeable to inform the developers of Newt editor (https://github.com/iVis-at-Bilkent/newt/issues/679).

An interesting field for future projects is definitely the integration of semantic annotations and functional aspects of the model into the SBGN map, with promising publications and reports from the community. As the approach in [2] to automatically create a first draft of SBGN was not suitable for the existing SBML files of the Bachmann model in our project, development of a guideline for modification of the SBML file to improve automated creation of SBGN would be very helpful.

References

[1] Bergmann, F.T. et al.. COMBINE archive and OMEX format: one file to share all information to reproduce a modeling project. BMC Bioinformatics 15, 369 (2014). https://doi.org/10.1186/s12859-014-0369-z
[2] Scharm, M. and Waltemath, D. A fully featured COMBINE archive of a simulation study on syncytial mitotic cycles in Drosophila embryos. F1000Research 5, 2421 (2016). https://doi.org/10.12688/f1000research.9379.1
[3] Bachmann, J. et al. Division of labor by dual feedback regulators controls JAK2/STAT5 signaling over broad ligand range. Molecular Systems Biology 7, 516 (2011). https://doi.org/10.1038/msb.2011.50