Merge pull request #4 from monarch-initiative/prepare-release

Updating to prepare for release 2022-05-27

README.md

@@ -1,20 +1,14 @@
+# Phenomics Integrated Ontology (PHENIO)
 
-![Build Status](https://github.com/monarch-initiative/phenio/workflows/CI/badge.svg)
-# Phenomics Integrated Ontology
-
-Description: None
-
-More information can be found at http://obofoundry.org/ontology/phenio
+An ontology for accessing and comparing knowledge concerning phenotypes across species and genetic backgrounds.
 
 ## Versions
 
 ### Stable release versions
 
 The latest version of the ontology can always be found at:
 
-http://purl.obolibrary.org/obo/phenio.owl
-
-(note this will not show up until the request has been approved by obofoundry.org)
+https://github.com/monarch-initiative/phenio/releases/
 
 ### Editors' version
 
@@ -26,4 +20,4 @@ Please use this GitHub repository's [Issue tracker](https://github.com/monarch-i
 
 ## Acknowledgements
 
-This ontology repository was created using the [Ontology Development Kit (ODK)](https://github.com/INCATools/ontology-development-kit).
+This ontology repository was created using the [Ontology Development Kit (ODK)](https://github.com/INCATools/ontology-development-kit).

docs/cite.md

@@ -0,0 +1 @@
+# How to cite PHENIO

docs/contributing.md

@@ -0,0 +1 @@
+# How to contribute to PHENIO

docs/history.md

@@ -0,0 +1,3 @@
+# A brief history of PHENIO
+
+The following page gives an overview of the history of PHENIO.

docs/index.md

@@ -0,0 +1,7 @@
+# PHENIO Ontology Documentation
+
+[//]: # "This file is meant to be edited by the ontology maintainer."
+
+Welcome to the PHENIO documentation!
+
+You can find descriptions of the standard ontology engineering workflows [here](odk-workflows/index.md).

docs/odk-workflows/ContinuousIntegration.md

@@ -0,0 +1,15 @@
+# Introduction to Continuous Integration Workflows with ODK
+
+Historically, most repos have been using Travis CI for continuous integration testing and building, but due to
+runtime restrictions, we recently switched a lot of our repos to GitHub actions. You can set up your repo with CI by adding 
+this to your configuration file (src/ontology/phenio-odk.yaml):
+
+```
+ci:
+  - github_actions
+```
+
+When [updateing your repo](RepoManagement.md), you will notice a new file being added: `.github/workflows/qc.yml`.
+
+This file contains your CI logic, so if you need to change, or add anything, this is the place!
+

docs/odk-workflows/EditorsWorkflow.md

@@ -0,0 +1,124 @@
+# Editors Workflow
+
+The editors workflow is one of the formal [workflows](index.md) to ensure that the ontology is developed correctly according to ontology engineering principles. There are a few different editors workflows:
+
+1. Local editing workflow: Editing the ontology in your local environment by hand, using tools such as Protégé, ROBOT templates or DOSDP patterns.
+2. Completely automated data pipeline (GitHub Actions)
+3. DROID workflow
+
+This document only covers the first editing workflow, but more will be added in the future
+
+### Local editing workflow
+
+Workflow requirements:
+- git
+- github
+- docker
+- editing tool of choice, e.g. Protégé, your favourite text editor, etc
+
+#### 1. _Create issue_
+Ensure that there is a ticket on your issue tracker that describes the change you are about to make. While this seems optional, this is a very important part of the social contract of building an ontology - no change to the ontology should be performed without a good ticket, describing the motivation and nature of the intended change.
+
+#### 2. _Update main branch_ 
+In your local environment (e.g. your laptop), make sure you are on the `main` (prev. `master`) branch and ensure that you have all the upstream changes, for example:
+
+```
+git checkout main
+git pull
+```
+
+#### 3. _Create feature branch_
+Create a new branch. Per convention, we try to use meaningful branch names such as:
+- issue23removeprocess (where issue 23 is the related issue on GitHub)
+- issue26addcontributor
+- release20210101 (for releases)
+
+On your command line, this looks like this:
+
+```
+git checkout -b issue23removeprocess
+```
+
+#### 4. _Perform edit_
+Using your editor of choice, perform the intended edit. For example:
+
+_Protege_
+
+1. Open `src/ontology/phenio-edit.owl` in Protege
+2. Make the change
+3. Save the file
+
+_TextEdit_
+
+1. Open `src/ontology/phenio-edit.owl` in TextEdit (or Sublime, Atom, Vim, Nano)
+2. Make the change
+3. Save the file
+
+Consider the following when making the edit.
+
+1. According to our development philosophy, the only places that should be manually edited are:
+    - `src/ontology/phenio-edit.owl`
+    - Any ROBOT templates you chose to use (the TSV files only)
+    - Any DOSDP data tables you chose to use (the TSV files, and potentially the associated patterns)
+    - components (anything in `src/ontology/components`), see [here](RepositoryFileStructure.md).
+2. Imports should not be edited (any edits will be flushed out with the next update). However, refreshing imports is a potentially breaking change - and is discussed [elsewhere](UpdateImports.md).
+3. Changes should usually be small. Adding or changing 1 term is great. Adding or changing 10 related terms is ok. Adding or changing 100 or more terms at once should be considered very carefully.
+
+#### 4. _Check the Git diff_
+This step is very important. Rather than simply trusting your change had the intended effect, we should always use a git diff as a first pass for sanity checking.
+
+In our experience, having a visual git client like [GitHub Desktop](https://desktop.github.com/) or [sourcetree](https://www.sourcetreeapp.com/) is really helpful for this part. In case you prefer the command line:
+
+```
+git status
+git diff
+```
+#### 5. Quality control
+Now its time to run your quality control checks. This can either happen locally ([5a](#Local-testing)) or through your continuous integration system ([7/5b](#continuous-integration-testing)).
+
+#### 5a. Local testing
+If you chose to run your test locally:
+
+```
+sh run.sh make IMP=false test
+```
+This will run the whole set of configured ODK tests on including your change. If you have a complex DOSDP pattern pipeline you may want to add `PAT=false` to skip the potentially lengthy process of rebuilding the patterns.
+
+```
+sh run.sh make IMP=false PAT=false test
+```
+
+#### 6. Pull request
+
+When you are happy with the changes, you commit your changes to your feature branch, push them upstream (to GitHub) and create a pull request. For example:
+
+```
+git add NAMEOFCHANGEDFILES
+git commit -m "Added biological process term #12"
+git push -u origin issue23removeprocess
+```
+
+Then you go to your project on GitHub, and create a new pull request from the branch, for example: https://github.com/INCATools/ontology-development-kit/pulls
+
+There is a lot of great advise on how to write pull requests, but at the very least you should:
+- mention the tickets affected: `see #23` to link to a related ticket, or `fixes #23` if, by merging this pull request, the ticket is fixed. Tickets in the latter case will be closed automatically by Github when the pull request is merged.
+- summarise the changes in a few sentences. Consider the reviewer: what would they want to know right away.
+- If the diff is large, provide instructions on how to review the pull request best (sometimes, there are many changed files, but only one important change).
+
+#### 7/5b. Continuous Integration Testing
+If you didn't run and local quality control checks (see [5a](#Local-testing)), you should have Continuous Integration (CI) set up, for example:
+- Travis
+- GitHub Actions
+
+More on how to set this up [here](ContinuousIntegration.md). Once the pull request is created, the CI will automatically trigger. If all is fine, it will show up green, otherwise red.
+
+#### 8. Community review
+Once all the automatic tests have passed, it is important to put a second set of eyes on the pull request. Ontologies are inherently social - as in that they represent some kind of community consensus on how a domain is organised conceptually. This seems high brow talk, but it is very important that as an ontology editor, you have your work validated by the community you are trying to serve (e.g. your colleagues, other contributors etc). In our experience, it is hard to get more than one review on a pull request - two is great. You can set up GitHub branch protection to actually require a review before a pull request can be merged! We recommend this.
+
+This step seems daunting to some hopefully under-resourced ontologies, but we recommend to put this high up on your list of priorities - train a colleague, reach out!
+
+#### 9. Merge and cleanup
+When the QC is green and the reviews are in (approvals), it is time to merge the pull request. After the pull request is merged, remember to delete the branch as well (this option will show up as a big button right after you have merged the pull request). If you have not done so, close all the associated tickets fixed by the pull request.
+
+#### 10. Changelog (Optional)
+It is sometimes difficult to keep track of changes made to an ontology. Some ontology teams opt to document changes in a changelog (simply a text file in your repository) so that when release day comes, you know everything you have changed. This is advisable at least for major changes (such as a new release system, a new pattern or template etc).

docs/odk-workflows/ManageDocumentation.md

@@ -0,0 +1,45 @@
+# Updating the Documentation
+
+The documentation for PHENIO is managed in two places (relative to the repository root):
+
+1. The `docs` directory contains all the files that pertain to the content of the documentation (more below)
+2. the `mkdocs.yaml` file cotains the documentation config, in particular its navigation bar and theme.
+
+The documentation is hosted using github pages, on a special branch of the repository (called `gh-pages`). It is important that this branch is never deleted - it contains all the files GitHub pages needs to render and deploy the site. It is also important to note that _the gh-pages branch should never be edited manually_. All changes to the docs happen inside the `docs` directory on the `main` branch.
+
+## Editing the docs
+
+### Changing content
+All the documentation is contained in the `docs` directory, and is managed in _Markdown_. Markdown is a very simple and convenient way to produce text documents with formatting instructions, and is very easy to learn - it is also used, for example, in GitHub issues. This is a normal editing workflow:
+
+1. Open the `.md` file you want to change in an editor of choice (a simple text editor is often best). _IMPORTANT_: Do not edit any files in the `docs/odk-workflows/` directory. These files are managed by the ODK system and will be overwritten when the repository is upgraded! If you wish to change these files, make an issue on the [ODK issue tracker](https://github.com/INCATools/ontology-development-kit/issues).
+2. Perform the edit and save the file
+3. Commit the file to a branch, and create a pull request as usual. 
+4. If your development team likes your changes, merge the docs into main branch.
+5. Deploy the documentation (see below)
+
+## Deploy the documentation
+
+The documentation is _not_ automatically updated from the Markdown, and needs to be deployed deliberately. To do this, perform the following steps:
+
+1. In your terminal, navigate to the edit directory of your ontology, e.g.:
+   ```
+   cd phenio/src/ontology
+   ```
+2. Now you are ready to build the docs as follows:
+   ```
+   sh run.sh make update_docs
+   ```
+   [Mkdocs](https://www.mkdocs.org/) now sets off to build the site from the markdown pages. You will be asked to
+    - Enter your username
+    - Enter your password (see [here](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token) for using GitHub access tokens instead)
+      _IMPORTANT_: Using password based authentication will be deprecated this year (2021). Make sure you read up on [personal access tokens](https://docs.github.com/en/github/authenticating-to-github/creating-a-personal-access-token) if that happens!
+
+   If everything was successful, you will see a message similar to this one:
+
+   ```
+   INFO    -  Your documentation should shortly be available at: https://monarch-initiative.github.io/phenio/ 
+   ```
+3. Just to double check, you can now navigate to your documentation pages (usually https://monarch-initiative.github.io/phenio/). 
+   Just make sure you give GitHub 2-5 minutes to build the pages!
+

docs/odk-workflows/ReleaseWorkflow.md

@@ -0,0 +1,69 @@
+# The release workflow 
+The release workflow recommended by the ODK is based on GitHub releases and works as follows:
+
+1. Run a release with the ODK
+2. Review the release
+3. Merge to main branch
+4. Create a GitHub release
+
+These steps are outlined in detail in the following.
+
+## Run a release with the ODK
+
+Preparation:
+
+1. Ensure that all your pull requests are merged into your main (master) branch
+2. Make sure that all changes to main are committed to Github (`git status` should say that there are no modified files)
+3. Locally make sure you have the latest changes from main (`git pull`)
+4. Checkout a new branch (e.g. `git checkout -b release-2021-01-01`)
+5. You may or may not want to refresh your imports as part of your release strategy (see [here](UpdateImports.md))
+6. Make sure you have the latest ODK installed by running `docker pull obolibrary/odkfull`
+
+To actually run the release, you:
+
+1. Open a command line terminal window and navigate to the src/ontology directory (`cd phenio/src/ontology`)
+2. Run release pipeline:`sh run.sh make prepare_release -B`. Note that for some ontologies, this process can take up to 90 minutes - especially if there are large ontologies you depend on, like PRO or CHEBI.
+3. If everything went well, you should see the following output on your machine: `Release files are now in ../.. - now you should commit, push and make a release on your git hosting site such as GitHub or GitLab`.
+
+This will create all the specified release targets (OBO, OWL, JSON, and the variants, ont-full and ont-base) and copy them into your release directory (the top level of your repo).
+
+## Review the release
+
+1. (Optional) Rough check. This step is frequently skipped, but for the more paranoid among us (like the author of this doc), this is a 3 minute additional effort for some peace of mind. Open the main release (phenio.owl) in you favourite development environment (i.e. Protege) and eyeball the hierarchy. We recommend two simple checks: 
+    1. Does the very top level of the hierarchy look ok? This means that all new terms have been imported/updated correctly.
+    2. Does at least one change that you know should be in this release appear? For example, a new class. This means that the release was actually based on the recent edit file. 
+2. Commit your changes to the branch and make a pull request
+3. In your GitHub pull request, review the following three files in detail (based on our experience):
+    1. `phenio.obo` - this reflects a useful subset of the whole ontology (everything that can be covered by OBO format). OBO format has that speaking for it: it is very easy to review!
+    2. `phenio-base.owl` - this reflects the asserted axioms in your ontology that you have actually edited.
+    3. Ideally also take a look at `phenio-full.owl`, which may reveal interesting new inferences you did not know about. Note that the diff of this file is sometimes quite large.
+4. Like with every pull request, we recommend to always employ a second set of eyes when reviewing a PR!
+
+## Merge the main branch
+Once your [CI checks](ContinuousIntegration.md) have passed, and your reviews are completed, you can now merge the branch into your main branch (don't forget to delete the branch afterwards - a big button will appear after the merge is finished).
+
+## Create a GitHub release
+
+1. Go to your releases page on GitHub by navigating to your repository, and then clicking on releases (usually on the right, for example: https://github.com/monarch-initiative/phenio/releases. Then click "Draft new release"
+1. As the tag version you **need to choose the date on which your ontologies were build.** You can find this, for example, by looking at the `phenio.obo` file and check the `data-version:` property. The date needs to be prefixed with a `v`, so, for example `v2020-02-06`.
+1. You can write whatever you want in the release title, but we typically write the date again. The description underneath should contain a concise list of changes or term additions.
+1. Click "Publish release". Done.
+
+## Debugging typical ontology release problems
+
+### Problems with memory
+
+When you are dealing with large ontologies, you need a lot of memory. When you see error messages relating to large ontologies such as CHEBI, PRO, NCBITAXON, or Uberon, you should think of memory first, see [here](https://github.com/INCATools/ontology-development-kit/blob/master/docs/DealWithLargeOntologies.md).
+
+### Problems when using OBO format based tools
+
+Sometimes you will get cryptic error messages when using legacy tools using OBO format, such as the ontology release tool (OORT), which is also available as part of the ODK docker container. In these cases, you need to track down what axiom or annotation actually caused the breakdown. In our experience (in about 60% of the cases) the problem lies with duplicate annotations (`def`, `comment`) which are illegal in OBO. Here is an example recipe of how to deal with such a problem:
+
+1. If you get a message like `make: *** [cl.Makefile:84: oort] Error 255` you might have a OORT error. 
+2. To debug this, in your terminal enter `sh run.sh make IMP=false PAT=false oort -B` (assuming you are already in the ontology folder in your directory) 
+3. This should show you where the error is in the log (eg multiple different definitions) 
+WARNING: THE FIX BELOW IS NOT IDEAL, YOU SHOULD ALWAYS TRY TO FIX UPSTREAM IF POSSIBLE
+4. Open `phenio-edit.owl` in Protege and find the offending term and delete all offending issue (e.g. delete ALL definition, if the problem was "multiple def tags not allowed") and save. 
+*While this is not idea, as it will remove all definitions from that term, it will be added back again when the term is fixed in the ontology it was imported from and added back in.
+5. Rerun `sh run.sh make IMP=false PAT=false oort -B` and if it all passes, commit your changes to a branch and make a pull request as usual.
+