Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run notebooks on RTD with sphinx-gallery and add WorkGraph tutorials #474

Open
wants to merge 39 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
1298f45
WIP: run notebooks on RTD
agoscinski Sep 11, 2024
cd0c49b
add deps
agoscinski Sep 11, 2024
8631643
add installation of req
agoscinski Sep 11, 2024
f25fe8e
introduce SPHINX_LINKCHECK to not execute for linkchecks, ignore quer…
agoscinski Sep 12, 2024
7430793
try to specify requirements in conda environment.yml
agoscinski Sep 12, 2024
4c204df
integrate workgraph tutorials
agoscinski Sep 12, 2024
db2e36a
debug linkcheck
agoscinski Sep 12, 2024
ddbc440
initialize sphinx-gallery
agoscinski Sep 12, 2024
1ea66fe
debug sphinx linkcheck
agoscinski Sep 12, 2024
4bdba12
fix warnings
agoscinski Sep 12, 2024
2b2c43e
need to build the sphinx doc because the index.rst references the aut…
agoscinski Sep 12, 2024
e52d86c
add nicer front page
agoscinski Sep 12, 2024
5b48209
add imports to conf.py
agoscinski Sep 12, 2024
8e67630
update conda version
agoscinski Sep 12, 2024
4db0691
pre-commit run
agoscinski Sep 12, 2024
1fa3aaf
use latest
agoscinski Sep 12, 2024
798304e
update CI run policy, it was run twice before
agoscinski Sep 12, 2024
c87d18f
maybe auto update solves libmamba solver issue
agoscinski Sep 12, 2024
adfd2c0
use minifore + mamba to see if libmambas solver is there
agoscinski Sep 12, 2024
86731fd
fix path
agoscinski Sep 12, 2024
102d5e8
add copy html files to setup
agoscinski Sep 12, 2024
42ff514
pre-commit run
agoscinski Sep 12, 2024
43f6116
downgrade to python 3.11 due to some conflicts
agoscinski Sep 12, 2024
12e2872
update README
agoscinski Sep 12, 2024
3a2c005
debug ci
agoscinski Sep 12, 2024
8049b8c
pre-commit run
agoscinski Sep 12, 2024
ef349ac
debug ci
agoscinski Sep 12, 2024
0a06c78
debug ci
agoscinski Sep 12, 2024
15fb57a
debug ci
agoscinski Sep 12, 2024
a4cac3a
debug ci
agoscinski Sep 12, 2024
f89a812
debug ci
agoscinski Sep 12, 2024
8eb5d0f
debug ci
agoscinski Sep 12, 2024
6118d30
debug ci
agoscinski Sep 12, 2024
e38a372
debug ci
agoscinski Sep 12, 2024
32bde7f
move linkcheck to RTD
agoscinski Sep 12, 2024
7620580
debug rtd
agoscinski Sep 12, 2024
e531860
add html to link ignore, also add_css_file
agoscinski Sep 12, 2024
55210f2
add regex correctly
agoscinski Sep 12, 2024
61ce7e6
Apply suggestions from code review
agoscinski Sep 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 8 additions & 24 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,11 @@
name: continuous-integration

on: [push, pull_request]
on:
push:
# only pushes to main trigger
branches: [main]
pull_request:
# always triggered
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change is unrelated to this PR. It stops that the CI is run two time for each PR. I will put this in a separate PR


jobs:

Expand All @@ -11,29 +16,8 @@ jobs:

steps:
- uses: actions/checkout@v2
- name: Set up Python 3.12
- name: Set up Python 3.11
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To build aiida-core in CI, I needed to use Python 3.11. For pre-commit it is not needed but I think we should just stick with Python 3.11 if there are issues with Python 3.12.

uses: actions/setup-python@v2
with:
python-version: 3.12
python-version: 3.11
- uses: pre-commit/[email protected]

build:
Copy link
Contributor Author

@agoscinski agoscinski Sep 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is now run in the RTD, check out the .readthedocs.yml


runs-on: ubuntu-latest
timeout-minutes: 30

steps:
- uses: actions/checkout@v2
- name: Set up Python 3.12
uses: actions/setup-python@v2
with:
python-version: 3.12
- name: Install python dependencies
run: |
pip install --upgrade pip
pip install -r requirements.txt
- name: "Build HTML docs"
run: |
make -C docs html linkcheck
env:
SPHINXOPTS: -nW --keep-going
22 changes: 18 additions & 4 deletions .readthedocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,21 @@ version: 2
build:
os: "ubuntu-22.04"
tools:
python: "3.12"
python: "miniconda3-3.12-24.1" # note that libmamba-solver is available since 22.1
jobs:
post_create_environment:
# - python -m pip install --exists-action=w --no-cache-dir -r requirements.txt
- rabbitmq-server -detached
- sleep 10
- rabbitmq-diagnostics status
- verdi presto
- verdi daemon start
- verdi status
- aiida-pseudo install sssp -x PBEsol
- verdi group list
- cat /proc/cpuinfo | grep processor | wc -l
- python -m sphinx -T --keep-going -b linkcheck -d docs/_build/doctrees -D language=en docs $READTHEDOCS_OUTPUT/html


# Build documentation in the docs/ directory with Sphinx
sphinx:
Expand All @@ -19,7 +33,7 @@ sphinx:
# See https://docs.readthedocs.io/en/latest/yaml-config.html#formats
formats: []

conda:
environment: environment.yml

# Optionally set the version of Python and requirements required to build your docs
python:
install:
- requirements: requirements.txt
10 changes: 6 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,17 +26,19 @@ If you have a question, feel free to just [open an issue](https://github.com/aii

### Prerequisites

* python 3.12 or greater
* python 3.11

### Build instructions

```bash
git clone https://github.com/aiidateam/aiida-tutorials.git
cd aiida-tutorials
pip install -r requirements.txt
conda env create --quiet --name aiida-tutorials --file environment.yml
conda activate aiida-tutorials
pre-commit install # enable pre-commit hooks (optional)
cd docs/
make
make -C docs html # to build docs
make -C docs linkcheck # to run link checks (only for dev)

# open build/html/index.html
```

Expand Down
93 changes: 85 additions & 8 deletions docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@
"sphinx_copybutton",
"sphinx_panels",
"sphinx_tabs.tabs",
"sphinx_gallery.gen_gallery",
]

myst_enable_extensions = [
Expand All @@ -68,7 +69,7 @@
ipython_mplbackend = ""

copybutton_selector = "div:not(.no-copy)>div.highlight pre"
copybutton_prompt_text = ">>> |\.\.\. |\$ |In \[\d*\]: | {2,5}\.\.\.: | {5,8}: "
copybutton_prompt_text = ">>> |... |$ |In [d*]: | {2,5}...: | {5,8}: "
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The escapes raised some warnings

copybutton_prompt_is_regexp = True

todo_include_todos = True
Expand Down Expand Up @@ -391,13 +392,12 @@
suppress_warnings = ["misc.highlighting_failure"]

# Links we ignore, because they do not work temporary and we cannot fix it
linkcheck_ignore = ["https://www.big-map.eu/"]


def setup(app):
"""Setup function called by sphinx."""
app.add_css_file("css/custom.css")

linkcheck_ignore = [
"https://www.big-map.eu/",
r".*concept/index.html",
r".*howto/index.html",
"http://127.0.0.1:8000/workgraph",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we ignore workgraph related files

]

# We are not installing a full aiida environment
nb_execution_mode = "off"
Expand All @@ -408,6 +408,29 @@ def setup(app):
"plumpy": ("https://plumpy.readthedocs.io/en/latest/", None),
}

gallery_src_relative_dir = "gallery" # relative path of the gallery src wrt. sphinx src

# we mimik the structure in the sphinx src directory in the gallery src directory
sphinx_src_autogen_dirs = ["sections/writing_workflows_with_workgraph/autogen"]

gallery_src_dirs = [
os.path.join(gallery_src_relative_dir, autogen_dir)
for autogen_dir in sphinx_src_autogen_dirs
] # path of the python scripts that should be executed
sphinx_gallery_conf = {
"filename_pattern": "/*",
"examples_dirs": gallery_src_dirs, # in sphinx-gallery doc referred as gallery source
"gallery_dirs": sphinx_src_autogen_dirs, # path to where to gallery puts generated files
}

# ignore in the autogenerated ipynb files to surpress warning
exclude_patterns.extend(
[
os.path.join(sphinx_src_autogen_dir, "*ipynb")
for sphinx_src_autogen_dir in sphinx_src_autogen_dirs
]
)

# Compile all things needed before building the docs
# For instance, convert the notebook templates to actual tutorial and solution versions
print(
Expand All @@ -416,3 +439,57 @@ def setup(app):
universal_newlines=True,
)
)

import shutil
from pathlib import Path


def copy_html_files(app, exception):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

copy paste from aiida-workgraph to move html files from the sphinx gallery folder to the sphinx source, so we can display the widgets

"""
Copy all .html files from source to build directory, maintaining the directory structure.
"""
print("Copying HTML files to build directory")
copy_print_info = "Copying HTML files to build directory"
print()
print(copy_print_info)
print(len(copy_print_info) * "=")
if exception is not None: # Only copy files if the build succeeded
print(
"Build failed, but we still try to copy the HTML files to the build directory"
)
try:
src_path = Path(app.builder.srcdir)
build_path = Path(app.builder.outdir)

copy_print_info = f"Copying html files from sphinx src directory {src_path}"
print()
print(copy_print_info)
print(len(copy_print_info) * "-")
for html_file in src_path.rglob("*.html"):
relative_path = html_file.relative_to(src_path)
destination_file = build_path / relative_path
destination_file.parent.mkdir(parents=True, exist_ok=True)
shutil.copy(html_file, destination_file)
print(f"Copy {html_file} to {destination_file}")

gallery_src_path = Path(app.builder.srcdir / Path(gallery_src_relative_dir))

copy_print_info = (
f"Copying html files from gallery src directory {gallery_src_path} to build"
)
print()
print(copy_print_info)
print(len(copy_print_info) * "-")
for html_file in gallery_src_path.rglob("*.html"):
relative_path = html_file.relative_to(gallery_src_path)
destination_file = build_path / relative_path
destination_file.parent.mkdir(parents=True, exist_ok=True)
shutil.copy(html_file, destination_file)
print(f"Copy {html_file} to {destination_file}")
except Exception as e:
print(f"Failed to copy HTML files: {e}")


def setup(app):
app.add_css_file("css/custom.css")
app.connect("build-finished", copy_html_files)
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
:orphan:

================================
Running workflows with WorkGraph
================================
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sphinx-gallery requires a GALLERY_HEADER.rst but we do not use if anywhere therefore the :orphan: at the top to surppres warnings

Loading
Loading