using mybinder.org, organization of docs and/or examples to new repo

[newville]

At the Developer Zoom meeting yesterday (22-March), we discussed a few topics. One was to migrate more example analyses to Jupyter notebooks and to put these onto mybinder.org (or maybe Google colab). A related discussion point was to consider moving the examples, and maybe to docs to a separate repository. Since mybinder.org points to a GitHub repo, these two topics are sort of related: we could move the docs and examples to one or two new repos ("xraylarch-examples", "xraylarch-docs", or joined at "xraylarch-examples-docs"), and then point mybinder.org to that.

I think I prefer keeping docs and examples together so that the docs can do a "literalinclude" of examples, and so to migrate docs and examples to the separate repo "xraylarch-examples-docs".

The current docs use the "gh-pages" branch from "xraypy/xraylarch". I would like to keep that, but that makes a slight change to "build and install latest docs" to be:

• go to xraylarch-examples-docs/docs, edit docs, and run make
• rsync docs/_build/html into the gh-pages branch of "xraylarch", and then publish with
  git commit -am "update docs" && git push

Any thoughts or preferences @maurov @Ameyanagi, anyone else?

[Ameyanagi]

@newville
I was thinking if git submodules would work.
https://git-scm.com/book/en/v2/Git-Tools-Submodules

We can make the doc and example as a separate repo, but linked to the xraylarch repo as a submodule.
When the people clone the repo, the submodule are not cloned to their local system, but they still can do so if they specify git clone --recurse-submodules.

By doing so, the documentation can still be builded in the xraylarch repo, but without messing up the commits. This also apply for examples.

[newville]

@Ameyanagi Thanks, yeah using a submodule might be worth doing. Just as an example of that, earlier I forgot that the tests run some of the examples. And this is a fine thing to expand -- we do not have enough unit or feature tests.

I have not had the greatest experience with submodules, but I have not tried since "subtree" came into existence, and I think this might be the right use case.

But if we put the examples (and so tests?) into a separate module or submodule, then you would need that submodule to run the tests.
Any actual development would then want to include all the submodules.

But, maybe this needs more thought. I think the basic problem is "repo is getting large". By themselves, the main folders are currently:

• larch/ 5 MB, mostly because we include binaries for feff etc for all 3 OSes in the code, for all OSes. If/when we add Feff10 and FDMNES, maybe this should be changed to an xraylarch_binaries package, with separate wheels per OS.
• doc/ 43 MB, mostly images.
• examples/ 83 MB, 32 MB of Jupyter files (no doubt dominated by the plotly plots), but also a few Larix Sessions that I just added (20MB).

So: there is some cleanup and reorganization to do. If we add more Jupyter notebooks, that folder could easily blow up. I am okay with not adding very many Larix sessions, but they are useful for testing/demo purposes.

Maybe we need multiple submodules.