Do not save source files in gh-pages branch after docs creation was: git repo too large

[felixcremer]

The size of the repository that is downloaded when we dev the repository to 500 MB which is taken quite long on a bad internet connection.

The docs are very large with a size of roughly 500 MB per doc build in the gh-pages branch. since 0.8.3 before that it was around 30MB.
I suspect the size increased by #487.
We also have some previews in the gh-pages branch laying around so that the overall size of the folder in the gh-pages branch is 11 GB uncompressed.

[rafaqz]

@lazarusA 11GB!!

whats going on 😅

[lazarusA]

git prune ? I suspect that is a bad makie svg output. Maybe we should set the default output to png,

CairoMakie.activate!(type = "png")

and definitely prune the repo.

[rafaqz]

Thanks, that makes sense.

Could we also just delete the whole docs branch and start again?

[lazarusA]

Could we also just delete the whole docs branch and start again?

Just go for it 😄 . I like your fearless approach 😆

[felixcremer]

Could we also just delete the whole docs branch and start again?

Wouldn't we then loose all old docs?

Could we also make sure, that we delete old preview folders? There are fifteen preview folders with 500mb each.

[rafaqz]

Do we care about the old docs?

I'm just not sure a prune will get all the svgs in question, and otherwise we will need to do one of those crazy git filter-branch operations to remove the specific problem files.

If you want to do it that would help 😅

[felixcremer]

If you want to do it that would help 😅

Definitely not tonight, lets see how annoyed I am going to be by the slow dev download in the next days. pop

[rafaqz]

I added a github action that does the clean up (I remembered someone added it to Interfaces.jl so copied that)

Now we can just delete any problem directories in gh_pages and it should remove all the history automatically on the next run.

I deleted the previews folder which should do most of it.

[felixcremer]

This definitively helped, but a freshly dev --local Raster still has a repo size of 481 MB.
i just looked into it and the main culprit for the size of the docs is that there are some RasterDataSources files in the gh-pages at
Rasters/dev/scripts/generated/basics/WorldClim.

I suspect that these are needed for the plotting, but not actually shown in the final docs.
Should we change this

RASTERDATASOURCES_PATH: "." in the Documenter.yml to /tmp so that these files are not added to the gh-pages branch?

[rafaqz]

Good idea. I'll also go through and manually delete those files from the 8.x docs

[rafaqz]

We should also clean up all the .nc, .tif and .shp files after plotting from them. Theyre much smaller but still some MBs

[felixcremer]

Changing the RASTERDATASOURCE path to tmp reduced the size of the docs folders on the gh-pages branch to 31 MB. But something is still bloating the git repo in general.

[rafaqz]

See above... but also there will be a lot of png files from plots

We need a line in the make.jl that deletes all nc, tif, shp etc extensions

[felixcremer]

Yes, but there are still some larger files in the gh-pages history that I would expect will be deleted by the new preview deletion action once you merge a PR that had preview. That would be a PR by yourself.

[rafaqz]

That should have happened by now? but there don't seem to be any previews turning up now, maybe the folder needs to be there?

[felixcremer]

The previews only show up on PRs that are from this repository and my PR was coming from my fork, because I don't have commit rights here.

[rafaqz]

I made a PR for the version bump hoping it would make a preview

[felixcremer]

Yes, and that also triggered the preview deletion. You can see that by the timestamp of the delete history commit in the gh-pages branch. But we didn't have a preview after my PR and after you deleted the RasterData sources from the 0.8.x docs in the gh-pages branch. I hope the situation will be much better after the next PR that is merged.
I think we will never have a preview folder in the gh-pages folder, because they will always be deleted upon merging the PR.