fixes for docs generation - missing links fixed with redirects

CHANGES.md

@@ -107,7 +107,7 @@
  The fix is to force `requests` to use the OS copy of the root cert bundle.
 
  Fix for this error:
- ```
+ ```console
  $ docker exec proxy python -c "import requests; requests.request('GET', 'https://lvupavicsmaster.ouranos.ca/geoserver')"
  Traceback (most recent call last):
  File "<string>", line 1, in <module>
@@ -123,13 +123,13 @@
  ```
 
  Default SSL root cert bundle of `requests`:
- ```
+ ```console
  $ docker exec proxy python -c "import requests; print requests.certs.where()"
  /usr/local/lib/python2.7/dist-packages/requests/cacert.pem
  ```
 
  Confirm the fix works:
- ```
+ ```console
  $ docker exec -it proxy bash
  root@37ed3a2a03ae:/opt/local/src/CanarieAPI/canarieapi# REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt python -c "import requests; requests.request('GET', 'https://lvupavicsmaster.ouranos.ca/geoserver')"
  root@37ed3a2a03ae:/opt/local/src/CanarieAPI/canarieapi#
@@ -714,7 +714,7 @@
  We have 4 Java component but only 1 is vulnerable: Thredds:
  
  **After fix**:
- ```
+ ```console
  $ docker run -it --rm unidata/thredds-docker:4.6.18 bash
  root@f65aadd2955c:/usr/local/tomcat# find -iname '**log4j**'
  ./webapps/thredds/WEB-INF/classes/log4j2.xml
@@ -725,7 +725,7 @@
  ```
  
  **Before fix (unidata/thredds-docker:4.6.15)**:
- ```
+ ```console
  $ docker exec -it thredds find / -iname '**log4j**'
  find: ‘/proc/1/map_files’: Operation not permitted
  find: ‘/proc/12/map_files’: Operation not permitted
@@ -740,7 +740,7 @@
  **Other components (ncwms2, geoserver, solr) have log4j older than version 2.0
  so supposedly not affected**:
  
- ```
+ ```console
  $ docker exec -it ncwms2 find / -iname '**log4j**'
  /opt/conda/envs/birdhouse/opt/apache-tomcat/webapps/ncWMS2/WEB-INF/classes/log4j.properties
  /opt/conda/envs/birdhouse/opt/apache-tomcat/webapps/ncWMS2/WEB-INF/lib/log4j-1.2.17.jar
@@ -1327,7 +1327,7 @@ Deployed to https://medus.ouranos.ca/jupyter/ for acceptance testing.
  database updates if such users exist on your server instance. To look for possible duplicates, the following command
  can be used. Duplicate entries must be updated or removed such that only unique emails are present.
 
- ```shell
+ ```console
  echo "select email,user_name from users" | \
  docker exec -i postgres-magpie psql -U $POSTGRES_MAGPIE_USERNAME magpiedb | \
  sort > /tmp/magpie_users.txt

Makefile

@@ -211,6 +211,69 @@ bump-check: ## Verifies that required bumpversion files are found
 version: ## Display project version
  @-$(MSG_I) "$(APP_NAME) version: $(APP_VERSION)"
 
+## --- Documentation targets --- ##
+
+define BROWSER_PYSCRIPT
+import os, webbrowser, sys
+try:
+ from urllib import pathname2url
+except:
+ from urllib.request import pathname2url
+
+webbrowser.open("file://" + pathname2url(os.path.abspath(sys.argv[1])))
+endef
+export BROWSER_PYSCRIPT
+override BROWSER_DISPLAY := python -c "$$BROWSER_PYSCRIPT"
+
+override DOC_ROOT := docs
+override DOC_INDEX := $(DOC_ROOT)/_build/html/index.html
+override DOC_XARGS ?= ## additional arguments for Sphinx build
+override DOC_XARGS := $(call clean_opt,$(DOC_XARGS))
+override DOC_DEBUG ?= ## set debug logging for Sphinx build
+override DOC_DEBUG := $(call clean_opt,$(DOC_DEBUG))
+ifneq ($(DOC_DEBUG),)
+ override DOC_XARGS := $(DOC_XARGS) -vv
+endif
+
+$(DOC_INDEX):
+ @-$(MSG_I) "Building docs..."
+ @$(SHELL) -c '$(CONDA_CMD) "$(MAKE)" -C "$(DOC_ROOT)" SPHINXOPTS="$(DOC_XARGS)" html;'
+ @-$(MSG_I) "Documentation available: [file://$(DOC_INDEX)]"
+
+.PHONY: _force_docs
+_force_docs:
+ @-rm -f "$(DOC_INDEX)"
+
+# rm without quotes important below to allow regex
+.PHONY: docs-clean
+docs-clean: ## remove doc artifacts
+ @-$(MSG_I) "Cleaning doc artifacts..."
+ @-find "$(DOC_ROOT)/" -type f -name "$(APP_NAME)*.rst" -delete
+ @-rm -f "$(DOC_ROOT)/modules.rst"
+ @-rm -rf "$(DOC_ROOT)/_build"
+
+.PHONY: docs-linkcheck
+docs-linkcheck: docs ## run check of external links in documentation for integrity
+ @-$(MSG_I) "Running link checks on docs..."
+ @$(SHELL) -c '$(CONDA_CMD) $(MAKE) -C "$(DOC_ROOT)" SPHINXOPTS="$(DOC_XARGS)" linkcheck'
+
+.PHONY: docs-install
+docs-install: ## install package requirements for documentation generation
+ @$(SHELL) -c '$(CONDA_CMD) pip install $(PIP_XARGS) -r "$(DOC_ROOT)/requirements.txt"'
+ @-$(MSG_I) "Successfully installed docs requirements."
+
+.PHONY: docs-only
+docs-only: _force_docs $(DOC_INDEX) ## generate documentation without requirements installation or cleanup
+
+# NOTE: we need almost all base dependencies because magpie package needs to be parsed to generate OpenAPI
+.PHONY: docs
+docs: docs-install docs-clean docs-only ## generate Sphinx HTML documentation
+
+.PHONY: docs-show
+docs-show: $(DOC_INDEX) ## display HTML webpage of generated documentation (build docs if missing)
+ @-test -f "$(DOC_INDEX)" || $(MAKE) -C "$(APP_ROOT)" $(DOC_ROOT)
+ $(BROWSER_DISPLAY) "$(DOC_INDEX)"
+
 ### Execution Targets ###
 
 SCRIPT ?= birdhouse/pavics-compose.sh ## Script to run the stack

README.rst

@@ -36,6 +36,13 @@ Power Analytics and Visualization for Climate Science - Powered by Birdhouse and
 
 For GitHub navigation, see the following README pages:
 
-* `README for general deployment <birdhouse/README.rst>`_
-* `README for extra core components <birdhouse/components/README.rst>`_
-* `README for optional components <birdhouse/optional-components/README.rst>`_
+* |readme_general|_
+* |readme_components|_
+* |readme_optional_components|_
+
+.. |readme_general| replace:: README for general deployment
+.. _readme_general: birdhouse/README.rst
+.. |readme_components| replace:: README for extra core components
+.. _readme_components: birdhouse/components/README.rst
+.. |readme_optional_components| replace:: README for optional components
+.. _readme_optional_components: birdhouse/optional-components/README.rst

birdhouse/README.rst

@@ -17,22 +17,29 @@ Requirements:
 * Install latest docker-ce and docker-compose for the chosen distro (not the
  version from the distro).
 
-To run ``docker-compose`` for PAVICS, the `pavics-compose.sh <pavics-compose.sh>`_ (:download:`download </birdhouse/pavics-compose.sh>`) wrapper script must be used.
-This script will source the ``env.local`` file, apply the appropriate variable substitutions on all the configuration files
-".template", and run ``docker-compose`` with all the command line arguments given to `pavics-compose.sh <pavics-compose.sh>`_ (:download:`download </birdhouse/pavics-compose.sh>`).
-See `env.local.example <env.local.example>`_ (:download:`download </birdhouse/env.local.example>`) for more details on what can go into the ``env.local`` file.
+To run ``docker-compose`` for PAVICS, the `pavics-compose.sh <pavics-compose.sh>`_
+(:download:`download </birdhouse/pavics-compose.sh>`) wrapper script must be used.
+This script will source the ``env.local`` file, apply the appropriate variable substitutions
+on all the configuration files ".template", and run ``docker-compose`` with all the command
+line arguments given to `pavics-compose.sh <pavics-compose.sh>`_ (:download:`download </birdhouse/pavics-compose.sh>`).
+See `env.local.example <env.local.example>`_ (:download:`download </birdhouse/env.local.example>`) for more details on
+what can go into the ``env.local`` file.
 
-If the file `env.local` is somewhere else, symlink it here, next to `docker-compose.yml <docker-compose.yml>`_ (:download:`download </birdhouse/docker-compose.yml>`) because many scripts assume this location.
+If the file `env.local` is somewhere else, symlink it here, next to `docker-compose.yml <docker-compose.yml>`_
+(:download:`download </birdhouse/docker-compose.yml>`) because many scripts assume this location.
 
 To follow infrastructure-as-code, it is encouraged to source control the above
 `env.local` file and any override needed to customized this PAVICS deployment
-for your organization. For an example of possible override, see how the `emu service <optional-components/emu/docker-compose-extra.yml>`_ (:download:`download </birdhouse/optional-components/emu/docker-compose-extra.yml>`)
-(`README <optional-components/README.rst#emu-wps-service-for-testing>`_) can be optionally added to the deployment via the `override mechanism <https://docs.docker.com/compose/extends/>`_.
-Ouranos specific override can be found in this `birdhouse-deploy-ouranos <https://github.com/bird-house/birdhouse-deploy-ouranos>`_ repo.
+for your organization. For an example of possible override,
+see how the `emu service <optional-components/emu/docker-compose-extra.yml>`_
+(:download:`download </birdhouse/optional-components/emu/docker-compose-extra.yml>`)
+(`README <birdhouse/optional-components/README.rst#emu-wps-service-for-testing>`_) can be optionally added to the
+deployment via the `override mechanism <https://docs.docker.com/compose/extends/>`_. Ouranos specific override
+can be found in this `birdhouse-deploy-ouranos <https://github.com/bird-house/birdhouse-deploy-ouranos>`_ repo.
 
 Suggested deployment layout:
 
-.. code-block::
+.. code-block:: text
 
  ├── birdhouse-deploy/ # this repo
  │   ├── birdhouse/
@@ -47,7 +54,8 @@ Suggested deployment layout:
 The automatic deployment is able to handle multiple repos, so will trigger if
 this repo or your private-personalized-config repo changes, giving you
 automated continuous deployment. See the continuous deployment setup section
-below and the variable ``AUTODEPLOY_EXTRA_REPOS`` in `env.local.example <env.local.example>`_ (:download:`download </birdhouse/env.local.example>`).
+below and the variable ``AUTODEPLOY_EXTRA_REPOS`` in `env.local.example <env.local.example>`_
+(:download:`download </birdhouse/env.local.example>`).
 
 The automatic deployment of the PAVICS platform, of the Jupyter tutorial
 notebooks and of the automatic deployment mechanism itself can all be
@@ -62,15 +70,17 @@ To launch all the containers, use the following command:
 
  ./pavics-compose.sh up -d
 
-If you get a ``'No applicable error code, please check error log'`` error from the WPS processes, please make sure that the WPS databases exists in the
-postgres instance. See `create-wps-pgsql-databases.sh <scripts/create-wps-pgsql-databases.sh>`_ (:download:`download </birdhouse/scripts/create-wps-pgsql-databases.sh>`).
+If you get a ``'No applicable error code, please check error log'`` error from the WPS processes, please
+make sure that the WPS databases exists in the postgres instance.
+See `create-wps-pgsql-databases.sh <scripts/create-wps-pgsql-databases.sh>`_
+(:download:`download </birdhouse/scripts/create-wps-pgsql-databases.sh>`).
 
 
 Note
 ----
 
 * All WPS requests should be completed within ``proxy_read_timeout`` of the
- Nginx proxy, see `nginx.conf`_ (:download:`download <birdhouse/config/proxy/nginx.conf>`).
+ Nginx proxy, see `nginx.conf`_ (:download:`download <birdhouse/config/proxy/nginx.conf.template>`).
  Any WPS requests that will take longer should use the async mode.
 
  Default value ``PROXY_READ_TIMEOUT_VALUE`` in `default.env`_ (:download:`download <birdhouse/default.env>`).
@@ -84,12 +94,12 @@ Manual steps post deployment
 Create public demo user in Magpie for JupyterHub login
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Use `create-magpie-users <scripts/create-magpie-users>`_ (:download:`download </birdhouse/scripts/create-magpie-users>`) or follow manual
-instructions below.
+Use `create-magpie-users <scripts/create-magpie-users>`_
+(:download:`download </birdhouse/scripts/create-magpie-users>`) or follow manual instructions below.
 
 ``config.yml`` file if using ``create-magpie-users``:
 
-.. code-block::
+.. code-block:: yaml
 
  users:
  - username: < value of JUPYTER_DEMO_USER in `env.local` >
@@ -121,7 +131,8 @@ https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests with pre-configured
 Jenkins at https://github.com/Ouranosinc/jenkins-config.
 
 For that test suite to pass, run the script
-`scripts/bootstrap-instance-for-testsuite <scripts/bootstrap-instance-for-testsuite>`_ (:download:`download </birdhouse/scripts/bootstrap-instance-for-testsuite>`)
+`scripts/bootstrap-instance-for-testsuite <scripts/bootstrap-instance-for-testsuite>`_
+(:download:`download </birdhouse/scripts/bootstrap-instance-for-testsuite>`)
 to prepare your new instance. Further documentation inside the script.
 
 Optional components
@@ -148,10 +159,12 @@ environment for testing or to have multiple flavors of PAVICS with slightly
 different combinations of the parts all running simultaneously in their
 respective VM, allowing us to see the differences in behavior.
 
-See `vagrant_variables.yml.example </vagrant_variables.yml.example>`_ (:download:`download </vagrant_variables.yml.example>`) for what's
+See `vagrant_variables.yml.example </vagrant_variables.yml.example>`_
+(:download:`download </vagrant_variables.yml.example>`) for what's
 configurable with Vagrant.
 
-If using Centos box, follow `disk-resize <vagrant-utils/disk-resize>`_ (:download:`download </birdhouse/vagrant-utils/disk-resize>`) after
+If using Centos box, follow `disk-resize <vagrant-utils/disk-resize>`_
+(:download:`download </birdhouse/vagrant-utils/disk-resize>`) after
 first ``vagrant up`` failure due to disk full. Then ``vagrant reload && vagrant
 provision`` to continue. If using Ubuntu box, no manual steps required,
 everything just works.
@@ -161,7 +174,7 @@ platform and the extension pack, and `Vagrant <https://www.vagrantup.com/downloa
 
 One time setup:
 
-.. code-block::
+.. code-block:: shell
 
  # Clone this repo and checkout the desired branch.
 
@@ -171,7 +184,7 @@ One time setup:
 
 Starting and managing the lifecycle of the VM:
 
-.. code-block::
+.. code-block:: shell
 
  # start everything, this is the only command needed to bring up the entire
  # PAVICS platform
@@ -310,7 +323,7 @@ Release Procedure
  * Run ``git push --tags`` to upload the new version.
 
 
-.. _nginx.conf: ./config/proxy/nginx.conf
-.. _default.env: ./default.env
+.. _nginx.conf: ../birdhouse/config/proxy/nginx.conf.template
+.. _default.env: ../birdhouse/default.env
 .. _`.bumpversion.cfg`: ../.bumpversion.cfg
 .. _CHANGES.md: ../CHANGES.md