Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Add minimal container for building docs #92

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

phlax
Copy link
Member

@phlax phlax commented Aug 5, 2020

I attempted to do a local build of docs - and it turned out to be a lot more complex than simply running ./docs/build.sh from the root of the envoy repo

I then tried with the envoyproxy/envoy-build image used in ci. This worked but the image is 3.6GB and the average run to build the docs took around 15 mins.

To try and make this simpler and less resource-painful, i created this recipe which creates the environment required for building the docs only.

The image works by either mounting an envoy repo in the host to /source or by setting the ENVOY_REPO env var for the container, in which case it will fetch the repo at startup

The bare image for the env was around 670MB - but as it was fetching/building bazel and pip deps on each run it still took around 15 mins.

I therefore made the recipe run the docs build script once during image build. This makes the image slightly larger ~1.66GB but the runtime for building docs with the resultant image is reduced to around 3.5mins

I have left the travis script in the PR. If you are interested in taking this code in principle i will remove it and hook up azure pipelines.

Signed-off-by: Ryan Northey <[email protected]>
Signed-off-by: Ryan Northey <[email protected]>
Signed-off-by: Ryan Northey <[email protected]>
Signed-off-by: Ryan Northey <[email protected]>
@lizan
Copy link
Member

lizan commented Aug 8, 2020

Related to #90, something we can do is build a image without C++ toolchains (i.e. only bazel and python), for RBE build front end, for RBE jobs all C++ build/linking is happening remote so it work in all RBE CI. That will be similar to what I did for GetEnvoy:

gcr.io/getenvoy-package/bazel-linux-glibc         72a26368093ecda6721b548f0a04dfad2d0ea51d   a30d7c2faad6        37 hours ago        960MB
gcr.io/getenvoy-package/rbe-linux-glibc           72a26368093ecda6721b548f0a04dfad2d0ea51d   8be68feaf9af        37 hours ago        2.68GB

I'm not a fan of warming bazel / pip caches in docker build phase for several reasons:

@phlax
Copy link
Member Author

phlax commented Aug 8, 2020

I'm not a fan of warming bazel / pip caches in docker build phase for several reasons:

the problem is that while removing the compiler makes the image smaller (as does using the official apt/debs from llvm), it doesnt make it any quicker to run build docs.

waiting 15 minutes to render small changes in docs is not a viable workflow for anyone focused on editing docs (3.5 mins is still pretty slow)

warming the caches afaict doesnt prevent newer versions being installed - certainly at least in the pip case - it just prevents you from downloading and recompiling deps that dont change very often, on every single build

@phlax
Copy link
Member Author

phlax commented Aug 8, 2020

the other option is to not warm the cache and then mount the caches in the image before running.

this has the advantage of keeping the image small, but it makes using it a load more complicated, and on first run - or lacking the proper mounts - very slow to use

@lizan
Copy link
Member

lizan commented Aug 8, 2020

the other option is to not warm the cache and then mount the caches in the image before running.

this has the advantage of keeping the image small, but it makes using it a load more complicated, and on first run - or lacking the proper mounts - very slow to use

We do mount in CI and default ci/run_envoy_docker.sh (by default it is /tmp/envoy-docker-build:/build, if doc build doesn't utilize that, then fix the script. All bazel caches are already in that mount path.

@lizan
Copy link
Member

lizan commented Aug 8, 2020

@phlax how did you run build docs? In my local box the after the second time the docs build is pretty quick:

ci/run_envoy_docker.sh 'ci/do_ci.sh docs'  0.10s user 0.10s system 0% cpu 1:14.94 total

A clean build in our CI (this doesn't include time to pull image, but include setting up bazel / pip cache) is about 5mins: https://app.circleci.com/pipelines/github/envoyproxy/envoy/31052/workflows/2c8a0668-0cba-4659-9609-d6f9acfd10a5/jobs/370762

@phlax
Copy link
Member Author

phlax commented Aug 9, 2020

We do mount in CI and default ci/run_envoy_docker.sh (by default it is /tmp/envoy-docker-build:/build, if doc build doesn't utilize that, then fix the script. All bazel caches are already in that mount path.

yep, as said, running the build inside that container works - its just a very large image

A clean build in our CI (this doesn't include time to pull image, but include setting up bazel / pip cache) is about 5mins: https://app.circleci.com/pipelines/github/envoyproxy/envoy/31052/workflows/2c8a0668-0cba-4659-9609-d6f9acfd10a5/jobs/370762

im afraid i cant access that. if its faster to compile i would guess that the build must be making use of the extra cores with circleci xlarge

here are timings on travis for 3 runs of building docs using the existing image and then using the image built in this pr

using ci/run_envoy_docker.sh

  • download image 102.34s
  • run1: 718.07s
  • run2: 117.79s
  • run3: 115.53s

using image from PR

  • download image: 51.00s
  • run1: 268.50s
  • run2: 263.22s
  • run3: 264.00s

the test run can be seen here https://travis-ci.org/github/phlax/envoy-docs/builds/716231495

not sure why this image is slower on subsequent builds - just caching is not speeding things up as much as i expected. Mounting the /build dir seems to be quicker than just warming caches - but i figure that can be done for this image too.

the point for me is that image size counts a lot. im working on several projects and regularly rmi images im not immediately working on. More often than not i avoid downloading/running anything locally and push tests to travis - so waiting for a large image to download is annoying if you are not using most of what is in it.

@lizan
Copy link
Member

lizan commented Aug 10, 2020

@phlax Right, so if you want to make a smaller image, don't warm bazel / pip cache because they are useless when /build is mounted. We don't really care travis perfomance because that's not where docs build are usually run.

Instead, if the goal is building a smaller image, do what I described in #92 (comment), and it can be used not only docs CI but all RBE based CI, i.e. release/gcc/asan/tsan, so the image is also tested. This helps your travis performance as well, why not?

@phlax phlax changed the title Add minimal container for building docs [WI{] Add minimal container for building docs Aug 10, 2020
@phlax phlax changed the title [WI{] Add minimal container for building docs [WIP] Add minimal container for building docs Aug 10, 2020
@phlax phlax marked this pull request as draft August 10, 2020 08:05
Base automatically changed from master to main January 15, 2021 23:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants