Rethinking the static web server for immutable infrastructure.
Container images has a build step. The old static file server requirements no longer apply. We can instead pre-process the directory structure to be hosted.
While other static file servers mount a volume, here we build a container that represents the content at a specific state. There is no longer a need to check the underlying file system for changes.
Problems that Kubernetes solves that web servers no longer need to solve:
- Scaling horizontally
Problems that Ingress and Envoy solves that web servers no longer need to solve:
- Path rewrites
- TLS termination
New needs when static content goes "immutable infrastructure":
- Predictable memory footprint
- Prometheus metrics export
See example Dockerfile.
The yolean/envoystatic:tooling-[gitref]
image is for the build step.
The yolean/envoystatic:[gitref]
is slightly more opinionated than the official envoy image.
- Sets loglevel and xDS names as default command.
- Uses subfolders to
/etc/envoyfor bootstrap and xDS to allow separate mounts if desired. - Runs as envoy's nonroot by default
- TODO distroless once envoy's distroless becomes multi-arch.
This project is a Proof-of-concept and not necessarily suitable for production. It explores if build time preparation of an HTTP server is preferrable to serving a directory as-is. The most notable caveat is that Envoy's warning on direct responses applies because we increase the limit to an arbitrarily high value.
By extension, stdlib: https://pkg.go.dev/mime#TypeByExtension
Detection: https://pkg.go.dev/github.com/gabriel-vasile/mimetype
TODO 304 responses on If-None-Match
TODO but we could obviously gzip assets at build time, preferrably per mime type.
We could compress at build time to save CPU at runtime. Content would contain both uncompressed and compressed files, to select based on header matching. The complexity might not be warranted compared to Envoy Compressor
Envoy was boarn a proxy: use a sidecar!
A lightweight form of dynamic content is to interpolate at response time. TODO verify that direct_response works with for example Lua or even Wasm.
If performance matters consider using a CDN.
For our use cases we actually only care about the ease of operations, but if anyone has benchmarks we're of course curious.
Requires ystack, RUNPLATFORM should be your cluster's os/arch:
RUNPLATFORM=linux/amd64
EXPORT_CACHE=false skaffold build --platform=$RUNPLATFORM --file-output=build.artifacts --cache-artifacts=false
# The job is required because we need to run specs in cluster; skaffold verify runs locally
kubectl delete job envoystatic-tests-html01-spec --ignore-not-found=true
skaffold deploy -a build.artifacts
skaffold verify -a build.artifacts
skaffold build --file-output=build.artifacts --cache-artifacts=false
kubectl delete job envoystatic-tests-html01-spec --ignore-not-found=true
skaffold deploy -a build.artifacts
skaffold verify -a build.artifacts
echo "# result images"
cat build.artifacts | jq -r '.builds | .[] | select(.imageName | contains("test") | not) | .tag'
To docker hub given TAG=
crane cp builds-registry.ystack.svc.cluster.local/yolean/envoystatic-tooling:$TAG yolean/envoystatic:tooling-$TAG
crane cp builds-registry.ystack.svc.cluster.local/yolean/envoystatic:$TAG yolean/envoystatic:$TAG
crane cp builds-registry.ystack.svc.cluster.local/yolean/envoystatic-debug:$TAG yolean/envoystatic:debug-$TAG
- Validate against RDS errors at runtime, at least in test. Ideally make them fatal.
- Adapt
max_direct_response_body_size_bytesto the size of the largest processed file. - Update bootstrap config according to deprecation warnings.
- A
skaffold devloop with output dir and route.yaml sync, for downstream work.- For example with
npm run buildin a Nextjs project
- For example with
- Favicon support
- Redirect to index.html per subdir
An alternative to inmemory, unless/until Envoy introduces on-request read from disk, could be to offload files to blob storage based on size and/or other attributes. It would be up to the build step to prepare the blob storage, and define proxy URL mapping. A sidecar using Minio could be used as PoC.