Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix doc missing links to repo files #261

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

fix doc missing links to repo files #261

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
605bf54
fixes for docs generation - missing links fixed with redirects
fmigneault Aug 25, 2022
c116c58
revert delete images
fmigneault Aug 26, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions CHANGES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,7 @@
The fix is to force `requests` to use the OS copy of the root cert bundle.

Fix for this error:
```
```console
$ docker exec proxy python -c "import requests; requests.request('GET', 'https://lvupavicsmaster.ouranos.ca/geoserver')"
Traceback (most recent call last):
File "<string>", line 1, in <module>
Expand All @@ -128,13 +128,13 @@
```

Default SSL root cert bundle of `requests`:
```
```console
$ docker exec proxy python -c "import requests; print requests.certs.where()"
/usr/local/lib/python2.7/dist-packages/requests/cacert.pem
```

Confirm the fix works:
```
```console
$ docker exec -it proxy bash
root@37ed3a2a03ae:/opt/local/src/CanarieAPI/canarieapi# REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt python -c "import requests; requests.request('GET', 'https://lvupavicsmaster.ouranos.ca/geoserver')"
root@37ed3a2a03ae:/opt/local/src/CanarieAPI/canarieapi#
Expand Down Expand Up @@ -719,7 +719,7 @@
We have 4 Java component but only 1 is vulnerable: Thredds:

**After fix**:
```
```console
$ docker run -it --rm unidata/thredds-docker:4.6.18 bash
root@f65aadd2955c:/usr/local/tomcat# find -iname '**log4j**'
./webapps/thredds/WEB-INF/classes/log4j2.xml
Expand All @@ -730,7 +730,7 @@
```

**Before fix (unidata/thredds-docker:4.6.15)**:
```
```console
$ docker exec -it thredds find / -iname '**log4j**'
find: ‘/proc/1/map_files’: Operation not permitted
find: ‘/proc/12/map_files’: Operation not permitted
Expand All @@ -745,7 +745,7 @@
**Other components (ncwms2, geoserver, solr) have log4j older than version 2.0
so supposedly not affected**:

```
```console
$ docker exec -it ncwms2 find / -iname '**log4j**'
/opt/conda/envs/birdhouse/opt/apache-tomcat/webapps/ncWMS2/WEB-INF/classes/log4j.properties
/opt/conda/envs/birdhouse/opt/apache-tomcat/webapps/ncWMS2/WEB-INF/lib/log4j-1.2.17.jar
Expand Down Expand Up @@ -1332,7 +1332,7 @@ Deployed to https://medus.ouranos.ca/jupyter/ for acceptance testing.
database updates if such users exist on your server instance. To look for possible duplicates, the following command
can be used. Duplicate entries must be updated or removed such that only unique emails are present.

```shell
```console
echo "select email,user_name from users" | \
docker exec -i postgres-magpie psql -U $POSTGRES_MAGPIE_USERNAME magpiedb | \
sort > /tmp/magpie_users.txt
Expand Down
63 changes: 63 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,69 @@ bump-check: ## Verifies that required bumpversion files are found
version: ## Display project version
@-$(MSG_I) "$(APP_NAME) version: $(APP_VERSION)"

## --- Documentation targets --- ##

define BROWSER_PYSCRIPT
import os, webbrowser, sys
try:
from urllib import pathname2url
except:
from urllib.request import pathname2url

webbrowser.open("file://" + pathname2url(os.path.abspath(sys.argv[1])))
endef
export BROWSER_PYSCRIPT
override BROWSER_DISPLAY := python -c "$$BROWSER_PYSCRIPT"

override DOC_ROOT := docs
override DOC_INDEX := $(DOC_ROOT)/_build/html/index.html
override DOC_XARGS ?= ## additional arguments for Sphinx build
override DOC_XARGS := $(call clean_opt,$(DOC_XARGS))
override DOC_DEBUG ?= ## set debug logging for Sphinx build
override DOC_DEBUG := $(call clean_opt,$(DOC_DEBUG))
ifneq ($(DOC_DEBUG),)
override DOC_XARGS := $(DOC_XARGS) -vv
endif

$(DOC_INDEX):
@-$(MSG_I) "Building docs..."
@$(SHELL) -c '$(CONDA_CMD) "$(MAKE)" -C "$(DOC_ROOT)" SPHINXOPTS="$(DOC_XARGS)" html;'
@-$(MSG_I) "Documentation available: [file://$(DOC_INDEX)]"

.PHONY: _force_docs
_force_docs:
@-rm -f "$(DOC_INDEX)"

# rm without quotes important below to allow regex
.PHONY: docs-clean
docs-clean: ## remove doc artifacts
@-$(MSG_I) "Cleaning doc artifacts..."
@-find "$(DOC_ROOT)/" -type f -name "$(APP_NAME)*.rst" -delete
@-rm -f "$(DOC_ROOT)/modules.rst"
@-rm -rf "$(DOC_ROOT)/_build"

.PHONY: docs-linkcheck
docs-linkcheck: docs ## run check of external links in documentation for integrity
@-$(MSG_I) "Running link checks on docs..."
@$(SHELL) -c '$(CONDA_CMD) $(MAKE) -C "$(DOC_ROOT)" SPHINXOPTS="$(DOC_XARGS)" linkcheck'

.PHONY: docs-install
docs-install: ## install package requirements for documentation generation
@$(SHELL) -c '$(CONDA_CMD) pip install $(PIP_XARGS) -r "$(DOC_ROOT)/requirements.txt"'
@-$(MSG_I) "Successfully installed docs requirements."

.PHONY: docs-only
docs-only: _force_docs $(DOC_INDEX) ## generate documentation without requirements installation or cleanup

# NOTE: we need almost all base dependencies because magpie package needs to be parsed to generate OpenAPI
.PHONY: docs
docs: docs-install docs-clean docs-only ## generate Sphinx HTML documentation

.PHONY: docs-show
docs-show: $(DOC_INDEX) ## display HTML webpage of generated documentation (build docs if missing)
@-test -f "$(DOC_INDEX)" || $(MAKE) -C "$(APP_ROOT)" $(DOC_ROOT)
$(BROWSER_DISPLAY) "$(DOC_INDEX)"

### Execution Targets ###

SCRIPT ?= birdhouse/pavics-compose.sh ## Script to run the stack
Expand Down
13 changes: 10 additions & 3 deletions README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,13 @@ Power Analytics and Visualization for Climate Science - Powered by Birdhouse and

For GitHub navigation, see the following README pages:

* `README for general deployment <birdhouse/README.rst>`_
* `README for extra core components <birdhouse/components/README.rst>`_
* `README for optional components <birdhouse/optional-components/README.rst>`_
* |readme_general|_
* |readme_components|_
* |readme_optional_components|_

.. |readme_general| replace:: README for general deployment
.. _readme_general: birdhouse/README.rst
.. |readme_components| replace:: README for extra core components
.. _readme_components: birdhouse/components/README.rst
.. |readme_optional_components| replace:: README for optional components
.. _readme_optional_components: birdhouse/optional-components/README.rst
59 changes: 36 additions & 23 deletions birdhouse/README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,22 +17,29 @@ Requirements:
* Install latest docker-ce and docker-compose for the chosen distro (not the
version from the distro).

To run ``docker-compose`` for PAVICS, the `pavics-compose.sh <pavics-compose.sh>`_ (:download:`download </birdhouse/pavics-compose.sh>`) wrapper script must be used.
This script will source the ``env.local`` file, apply the appropriate variable substitutions on all the configuration files
".template", and run ``docker-compose`` with all the command line arguments given to `pavics-compose.sh <pavics-compose.sh>`_ (:download:`download </birdhouse/pavics-compose.sh>`).
See `env.local.example <env.local.example>`_ (:download:`download </birdhouse/env.local.example>`) for more details on what can go into the ``env.local`` file.
To run ``docker-compose`` for PAVICS, the `pavics-compose.sh <pavics-compose.sh>`_
(:download:`download </birdhouse/pavics-compose.sh>`) wrapper script must be used.
This script will source the ``env.local`` file, apply the appropriate variable substitutions
on all the configuration files ".template", and run ``docker-compose`` with all the command
line arguments given to `pavics-compose.sh <pavics-compose.sh>`_ (:download:`download </birdhouse/pavics-compose.sh>`).
See `env.local.example <env.local.example>`_ (:download:`download </birdhouse/env.local.example>`) for more details on
what can go into the ``env.local`` file.

If the file `env.local` is somewhere else, symlink it here, next to `docker-compose.yml <docker-compose.yml>`_ (:download:`download </birdhouse/docker-compose.yml>`) because many scripts assume this location.
If the file `env.local` is somewhere else, symlink it here, next to `docker-compose.yml <docker-compose.yml>`_
(:download:`download </birdhouse/docker-compose.yml>`) because many scripts assume this location.

To follow infrastructure-as-code, it is encouraged to source control the above
`env.local` file and any override needed to customized this PAVICS deployment
for your organization. For an example of possible override, see how the `emu service <optional-components/emu/docker-compose-extra.yml>`_ (:download:`download </birdhouse/optional-components/emu/docker-compose-extra.yml>`)
(`README <optional-components/README.rst#emu-wps-service-for-testing>`_) can be optionally added to the deployment via the `override mechanism <https://docs.docker.com/compose/extends/>`_.
Ouranos specific override can be found in this `birdhouse-deploy-ouranos <https://github.com/bird-house/birdhouse-deploy-ouranos>`_ repo.
for your organization. For an example of possible override,
see how the `emu service <optional-components/emu/docker-compose-extra.yml>`_
Copy link
Collaborator

@tlvu tlvu Sep 1, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's the link that did not resolve on RtD (https://birdhouse-deploy--261.org.readthedocs.build/en/261/optional-components/emu/docker-compose-extra.yml)

(:download:`download </birdhouse/optional-components/emu/docker-compose-extra.yml>`)
(`README <birdhouse/optional-components/README.rst#emu-wps-service-for-testing>`_) can be optionally added to the
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one resolve fine (https://birdhouse-deploy--261.org.readthedocs.build/en/261/birdhouse/optional-components/README.rst#emu-wps-service-for-testing)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't work in different manners per client/browser.
In Chrome (in my case at least), it shows the raw code of the HTML instead of displaying it.
In Firefox, it simply downloads the file.

deployment via the `override mechanism <https://docs.docker.com/compose/extends/>`_. Ouranos specific override
can be found in this `birdhouse-deploy-ouranos <https://github.com/bird-house/birdhouse-deploy-ouranos>`_ repo.

Suggested deployment layout:

.. code-block::
.. code-block:: text

├── birdhouse-deploy/ # this repo
│   ├── birdhouse/
Expand All @@ -47,7 +54,8 @@ Suggested deployment layout:
The automatic deployment is able to handle multiple repos, so will trigger if
this repo or your private-personalized-config repo changes, giving you
automated continuous deployment. See the continuous deployment setup section
below and the variable ``AUTODEPLOY_EXTRA_REPOS`` in `env.local.example <env.local.example>`_ (:download:`download </birdhouse/env.local.example>`).
below and the variable ``AUTODEPLOY_EXTRA_REPOS`` in `env.local.example <env.local.example>`_
(:download:`download </birdhouse/env.local.example>`).

The automatic deployment of the PAVICS platform, of the Jupyter tutorial
notebooks and of the automatic deployment mechanism itself can all be
Expand All @@ -62,15 +70,17 @@ To launch all the containers, use the following command:

./pavics-compose.sh up -d

If you get a ``'No applicable error code, please check error log'`` error from the WPS processes, please make sure that the WPS databases exists in the
postgres instance. See `create-wps-pgsql-databases.sh <scripts/create-wps-pgsql-databases.sh>`_ (:download:`download </birdhouse/scripts/create-wps-pgsql-databases.sh>`).
If you get a ``'No applicable error code, please check error log'`` error from the WPS processes, please
make sure that the WPS databases exists in the postgres instance.
See `create-wps-pgsql-databases.sh <scripts/create-wps-pgsql-databases.sh>`_
(:download:`download </birdhouse/scripts/create-wps-pgsql-databases.sh>`).


Note
----

* All WPS requests should be completed within ``proxy_read_timeout`` of the
Nginx proxy, see `nginx.conf`_ (:download:`download <birdhouse/config/proxy/nginx.conf>`).
Nginx proxy, see `nginx.conf`_ (:download:`download <birdhouse/config/proxy/nginx.conf.template>`).
Any WPS requests that will take longer should use the async mode.

Default value ``PROXY_READ_TIMEOUT_VALUE`` in `default.env`_ (:download:`download <birdhouse/default.env>`).
Expand All @@ -84,12 +94,12 @@ Manual steps post deployment
Create public demo user in Magpie for JupyterHub login
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Use `create-magpie-users <scripts/create-magpie-users>`_ (:download:`download </birdhouse/scripts/create-magpie-users>`) or follow manual
instructions below.
Use `create-magpie-users <scripts/create-magpie-users>`_
(:download:`download </birdhouse/scripts/create-magpie-users>`) or follow manual instructions below.

``config.yml`` file if using ``create-magpie-users``:

.. code-block::
.. code-block:: yaml

users:
- username: < value of JUPYTER_DEMO_USER in `env.local` >
Expand Down Expand Up @@ -121,7 +131,8 @@ https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests with pre-configured
Jenkins at https://github.com/Ouranosinc/jenkins-config.

For that test suite to pass, run the script
`scripts/bootstrap-instance-for-testsuite <scripts/bootstrap-instance-for-testsuite>`_ (:download:`download </birdhouse/scripts/bootstrap-instance-for-testsuite>`)
`scripts/bootstrap-instance-for-testsuite <scripts/bootstrap-instance-for-testsuite>`_
(:download:`download </birdhouse/scripts/bootstrap-instance-for-testsuite>`)
to prepare your new instance. Further documentation inside the script.

Optional components
Expand All @@ -148,10 +159,12 @@ environment for testing or to have multiple flavors of PAVICS with slightly
different combinations of the parts all running simultaneously in their
respective VM, allowing us to see the differences in behavior.

See `vagrant_variables.yml.example </vagrant_variables.yml.example>`_ (:download:`download </vagrant_variables.yml.example>`) for what's
See `vagrant_variables.yml.example </vagrant_variables.yml.example>`_
(:download:`download </vagrant_variables.yml.example>`) for what's
configurable with Vagrant.

If using Centos box, follow `disk-resize <vagrant-utils/disk-resize>`_ (:download:`download </birdhouse/vagrant-utils/disk-resize>`) after
If using Centos box, follow `disk-resize <vagrant-utils/disk-resize>`_
(:download:`download </birdhouse/vagrant-utils/disk-resize>`) after
first ``vagrant up`` failure due to disk full. Then ``vagrant reload && vagrant
provision`` to continue. If using Ubuntu box, no manual steps required,
everything just works.
Expand All @@ -161,7 +174,7 @@ platform and the extension pack, and `Vagrant <https://www.vagrantup.com/downloa

One time setup:

.. code-block::
.. code-block:: shell

# Clone this repo and checkout the desired branch.

Expand All @@ -171,7 +184,7 @@ One time setup:

Starting and managing the lifecycle of the VM:

.. code-block::
.. code-block:: shell

# start everything, this is the only command needed to bring up the entire
# PAVICS platform
Expand Down Expand Up @@ -313,7 +326,7 @@ Release Procedure
* Run ``git push --tags`` to upload the new version.


.. _nginx.conf: ./config/proxy/nginx.conf
.. _default.env: ./default.env
.. _nginx.conf: ../birdhouse/config/proxy/nginx.conf.template
.. _default.env: ../birdhouse/default.env
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why go up one level then back in birdhouse/ since the file is already in birdhouse/?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fancy trick to detect the nested birdhouse linked under docs/bidrhouse when building docs since the root of the source changes.

.. _`.bumpversion.cfg`: ../.bumpversion.cfg
.. _CHANGES.md: ../CHANGES.md
Loading