[Feature Request] Build only latest version / unmodified ones

[fraimondo]

I use sphinx-multiversion to keep track of the documentation history so users can read the documentation that matches their version of the software. However, after a few releases, Travis-CI is taking quite some time to build since it has to checkout every version of the doc and build it. Furthermore, it happens that the examples (sphinx-gallery) are ran against the latest version of the library.

It would be great if sphinx-multiversion can somehow do incremental builds as sphinx-build. So in this case, in the Travis-CI/GitHub Pages integration, we could check-out the current GitHub Pages, build on top of it (so it will only update certain files) and then push the updated documentation.

Great package by the way! Just trying to optimise resources here.

[gilesknap]

Hi, I believe I have addressed this issue in #64. Maybe you would like to give it a try and comment on the PR?

[fraimondo]

I've just went through the PR, and I think that will solve the issue (partially).

If I understood correctly, you clone the repo to a temporary folder and build the doc for each tag/ref using sphinx in a different folder.

This raises two issues:
1- If you land in an old tag, you might think that it is the latest (release). Or maybe that one and then main.
2- The examples that are built with sphinx-gallery are build using the latest master/main as there is no way of telling smv to install a specific version in the virtual environment before building.

Issue 1 is (partially) solved by always keeping master branch. Although it would be ideal to have some sort of selector that works using javascript and reads the possible tags from a remote json that it's updated with every build. From the plugin side, it's just as spitting a json file in the root with a predetermined format. Then the rest is on the templates, which instead of adding it as html, it can be added as javascript that requests the json file and populates accordingly.

Additionally, it will need some extra-setup when you use it within CI as GithubPages, as you will need to checkout the gh-pages branch in the output dir

Issue 2 is more problematic. Not too much if you set it up as CI and every time you add a tag is gets built with the latest master/main which at that point in time corresponds to the tag. Nevertheless, I think it's worth documenting the behaviour so users realise that the code is always run on the same version of the virtual environment, that it's just the doc versions that changes.

In any case, it's a great plugin and a great start to the issue. I'm just worried about the end-users of my library who might not be aware of the versions of the library. They consider master/main is the development one and if they see only one tag before that, then that would be latest stable.

[gilesknap]

Thanks for the feedback.
1 - this is true, but I believe a necessary evil since Sphinx embeds the index in every page. So a full rebuild is required if you want old versions to have newer version links. To get around this would involve significant changes to the internals of Sphinx as I understand it.
2 - I admit I have built this very much with CI in mind rather than local use. See here for a working example of github actions using this to deploy to github pages https://github.com/PandABlocks/PandABlocks-client/blob/master/.github/workflows/build.yml . I agree that using this in a local build may not update master with the new link so that is not ideal.

[daxpryce]

Hi! I'm basically looking for this functionality too - our project currently has a ton of jupyter notebooks we run through to generate a suite of tutorial documentation for graspologic. Currently our notebooks are breaking readthedocs' runtime limits on memory consumption - something that's way less of a concern in a Github action.

Rebuilding all of the documents for every published version would be ~13 mins per release, which is... going to be pretty brutal, after only a few releases.

Using RTD as an example, I know it doesn't rebuild all of our documentation from scratch when it gives us a drop-down selector of updated versions, but it's clearly doing something to support this. I would have to imagine it's something like what @fraimondo suggested - an ancillary artifact in addition to the documentation, probably json (or something) that gets output to a specific, fixed location on maybe the gh-pages branch, and all of the various versions are just html and JS that always know to look there to grab the latest, right? I can't imagine how else they'd manage it, at any rate. I'm sure there's a ton of race conditions to be concerned with there, though, so I'm not sure what the best approach is, really.

I suppose you can just consider this a long-winded 👍 to the request!

(And if you have any ideas on how to use sphinx-multiversion to solve this type of problem, I'm all ears!)

[fraimondo]

I integrated this solution into my package: https://github.com/juaml/julearn/

Now the existing tags are not built. However, I still have one issue. When there is a pull request, docs are not built. You can check here the output: https://github.com/juaml/julearn/runs/1786309367?check_suite_focus=true

My guess is that since it is not the main/master branch, it is not being built. This is because my smv_branch_whitelist is set to main. However I would like that the branch being checked out is built, to test if the documentation builds properly before merging the PR.

I just made a simple local test, and indeed it builds main + tags (skipped if already present). If I remove smv_branch_whitelist, it builds all the branches. I would like to achieve the behaviour where the current branch + tags are build.

How can I achieve this? Can I override the whitelist somehow in run time?

I'm trying something like -D 'smv_branch_whitelist=<BRANCH>' in the command line, but I get this:

unknown config value 'smv_branch_whitelist' in override, ignoring
unknown config value 'smv_branch_whitelist' in override, ignoring
unknown config value 'smv_branch_whitelist' in override, ignoring
unknown config value 'smv_branch_whitelist' in override, ignoring
unknown config value 'smv_branch_whitelist' in override, ignoring
unknown config value 'smv_branch_whitelist' in override, ignoring
unknown config value 'smv_branch_whitelist' in override, ignoring

[Holzhaus]

For pull requests I usually just build that particular branch without sphinx-multiversion. When reviewing aPR I'm not interested in other branches/tags. This also saves build time on the CI servers.

[gilesknap]

@fraimondo my project does build the docs on a pull request. For example see https://github.com/PandABlocks/PandABlocks-client/pull/13/checks?check_run_id=1550010895 . A PR does two builds, one against the source branch and one against the test merge to master. The latter does cause sphinx-multiversion with branch_whitelist 'master' and rebuild_tags=False to build the docs in my experience.

I've taken a quick look at your project and can't immediately see why you are not getting the same result.

'unknown config value' seems to imply I have not fully implemented smv_branch_whitelist. @Holzhaus can you comment on this please?

[Holzhaus]

If I remove smv_branch_whitelist, it builds all the branches.
[...]
I'm trying something like -D 'smv_branch_whitelist=<BRANCH>' in the command line, but I get this:

unknown config value 'smv_branch_whitelist' in override, ignoring

You can't override a value that is not present in the sphinx config. Why not set it to ".*" ?

[enricosutera]

Hi there, I'm also trying to build a single branch. I've tried to set smv_branch_whitelist to ".*" (also to None, r'^.*$') in conf.py... I always get the same line:
unknown config value 'smv_branch_whitelist' in override, ignoring
I'd like to add some logic to build only current branch in a pipeline, but for some reason also some other branches are built.

I've tried:
sphinx-multiversion -D "smv_branch_whitelist=$(git branch --show-current)" . build/html with this result:

unknown config value 'smv_branch_whitelist' in override, ignoring
unknown config value 'smv_branch_whitelist' in override, ignoring
unknown config value 'smv_branch_whitelist' in override, ignoring
unknown config value 'smv_branch_whitelist' in override, ignoring
...

For some reason the current branch is built but also some other with completely different name are.
Also I get outputdir '<BRANCH>' for refs/remotes/origin/<BRANCH> conflicts with other versions
Am i missing something?

[enricosutera]

The other built branch were actually tags.. so it is resolved using

sphinx-multiversion -D "smv_branch_whitelist=$(git branch --show-current)" -D "smv_tag_whitelist=$(git branch --show-current)" . build/html

I'm using read the docs template, however using this override, the index is populated with the current branch only. Is there any way of having all branches in the sidebar?

The template:

{%- if current_version %}
<div class="rst-versions" data-toggle="rst-versions" role="note" aria-label="versions">
  <span class="rst-current-version" data-toggle="rst-current-version">
    <span class="fa fa-book"> Other Versions</span>
    v: {{ current_version.name }}
    <span class="fa fa-caret-down"></span>
  </span>
  <div class="rst-other-versions">
    {%- if versions.tags %}
    <dl>
      <dt>Tags</dt>
      {%- for item in versions.tags %}
      <dd><a href="{{ item.url }}">{{ item.name }}</a></dd>
      {%- endfor %}
    </dl>
    {%- endif %}
    {%- if versions.branches %}
    <dl>
      <dt>Branches</dt>
      {%- for item in versions.branches %}
      <dd><a href="{{ item.url }}">{{ item.name }}</a></dd>
      {%- endfor %}
    </dl>
    {%- endif %}
  </div>
</div>
{%- endif %}