miniaturs

Tiny HTTP image resizer.

Goals

• Secure
• Fast:
  • Startup should be in the low 2-digit ms range (e.g., avoid "oh, it's a lambda")
  • Processing should be quick
• Cheap:
  • Pay as little as possible and avoid unnecessary work
  • Being fast can help minimise costs
• Scalable: can handle lots of requests
• Thumbor-ish
• Fetch from remote urls, but be good net citizen (don’t make requests to third parties if we have it in cache)
• Debuggable

To fulfil the above:

• Runs in a Lambda
  • Running on ARM, because it's cheaper: "about 34% cheaper compared to the default x86 processors"
• Rust ⚡️
• Caching in layers: CDN to protect the app, with S3 for storing images
• Serverless, but built on HTTP framework (cargo-lambda on top of axum)

An example Terraform config in terraform/prod is provided to show how to deploy at a subdomain using Cloudflare as our (free!) CDN + WAF with AWS Lambda and S3 (also free!)

Usage:

We only support resizing at the moment

1. An "image" endpoint a la Thumbor
   • GET /{HMAC_signature}/-Wx-H/{image_url}
2. A "metadata" endpoint a la Thumbor
   • GET /{HMAC_signature}/meta/-Wx-H/{image_url}
   • Difference: target image size is not returned (might change in the future)

Confguration

miniaturs relies on environment variables for configuration. These include

• MINIATURS_SHARED_SECRET : required, used for signature verification
• UNPROCESSED_IMAGES_BUCKET : required, bucket used for caching unprocessed images
• PROCESSED_IMAGES_BUCKET : required, bucket used for caching processed images
• REQUIRE_PATH_STYLE_S3 : optional, whether to use "path style" S3 addressing (for local testing), defaults to false.
• MAX_RESIZE_TARGET_WIDTH : optional, max resize-to image width, defaults to 10,000 (pixels)
• MAX_RESIZE_TARGET_HEIGHT : optional, max resize-to image height, defaults to 10,000 (pixels)
• MAX_SOURCE_IMAGE_WIDTH : optional, max source image width, defaults to 10,000 (pixels)
• MAX_SOURCE_IMAGE_HEIGHT : optional, max source image height, defaults to 10,000 (pixels)
• MAX_IMAGE_DOWNLOAD_SIZE : optional, max source image download size (as reported by content-length header), defaults to 10mb (must be parseable by bytesize)
• MAX_IMAGE_FILE_SIZE : optional, max source (post-download) image size, defaults to 10mb (must be parseable by bytesize)

Flow

Flow for an image resize request. The CDN/WAF graph is included since it's an important part of keeping costs low (and included in the prod terraform example), but is not a must.

flowchart
   imageReq(("Image request"))


   respEnd((Ok/Error response))

   imageReq --> cdnReq
   cdnResp --> respEnd

   subgraph CDN/WAF
      cdnReq[\"Request"\]
      cdnResp[\"Ok/Error Response"\]
      cdnCache[("CDN Cache")]
      wafBlocked{"Rate limited?"}
      cdnCached{"Cached?"}
      cache["Cache"]

      cdnReq --> wafBlocked
      wafBlocked -->|yes| cdnResp

      wafBlocked -->|no| cdnCached

      cdnCached <-->|fetch| cdnCache
      cdnCached -->|yes| cdnResp


      cache -->|cache| cdnCache
      cache --> cdnResp


   end

   subgraph Lambda

      lambdaRequest[\"Request"\]
      sigCheck{"Signature OK"?}
      toOps[To Operations]
      opsCheck{"Operations OK?"}

      processedCache[("Processed cache")]
      fetchCachedResult["Fetch cached result"]
      opsResultCached{"Operations Result Cached?"}


      rawCache[("Unprocessed cache")]
      fetchCachedRaw["Fetch cached unprocessed image"]
      rawCached{"Unprocessed image Cached?"}

      fetchRaw["Fetch unprocessed image from remote"]
      rawImageValidation{"Unprocessed image validations pass?"}
      cacheRaw["Cache unprocessed image"]

      processImage["Process image according to operations"]
      cacheResult["Cache processed result"]
      lambdaResponse[\"Ok/Error Response"\]


      cdnCached -->|no| lambdaRequest
      lambdaRequest --> sigCheck
      sigCheck -->|no| lambdaResponse
      sigCheck -->|yes| toOps

      toOps --> opsCheck
      opsCheck -->|no| lambdaResponse
      opsCheck -->|yes| fetchCachedResult
      fetchCachedResult <-->|fetch| processedCache
      fetchCachedResult --> opsResultCached
      opsResultCached -->|yes| lambdaResponse

      opsResultCached -->|no| fetchCachedRaw
      fetchCachedRaw <-->|fetch| rawCache
      fetchCachedRaw --> rawCached
      rawCached -->|no| fetchRaw
      fetchRaw --> rawImageValidation
      rawImageValidation -->|no| lambdaResponse
      rawImageValidation -->|yes| cacheRaw
      cacheRaw -->|cache| rawCache

      cacheRaw --> processImage
      rawCached -->|yes| processImage

      processImage --> cacheResult
      cacheResult -->|cache| processedCache

      cacheResult --> lambdaResponse
      lambdaResponse --> cache
   end

Loading

Development

Rust

Assuming we have the Rust toolbelt installed, the main thing we need is cargo-lambda

❯ brew tap cargo-lambda/cargo-lambda
❯ brew install cargo-lambda

AWS

• brew install awscli to install the CLI
• Log into your app

Ensure:

• aws configure sso is done
• .aws/config has the correct profile configuration, with a [profile ${PROFILE_NAME}] line where PROFILE_NAME matches what is in main.tf

Login for Terraform

aws sso login --profile ${PROFILE_NAME}

Cloudflare

Ensure CLOUDFLARE_API_TOKEN is defined in the environment (needed for Cloudflare provider and cache busting). It’ll need privileges for updating DNS and cache settings.

Deploying

Terraform

• Use tfenv: https://formulae.brew.sh/formula/tfenv

• Check what version is needed and install using ^

• For local dev, localstack is used (see terraform/localdev/docker-compose.yaml), and tflocal is used (https://formulae.brew.sh/formula/terraform-local)

  • docker-compose through official Docker or Rancher is supported, but enabling admin access is needed for running tests with Rancher

Per Environment

Use Makefile targets.

• For local dev:
  • make init_dev_env (only needed if TF complains)
  • make start_dev_env provision_dev_env
  • make begin_dev
  • TO_SIGN="200x-100/https://beachape.com/images/octopress_with_container.png" make signature_for_localstack to get a signed path for dev env
  • TO_SIGN="200x-100/https://beachape.com/images/octopress_with_container.png" make signature_for_dev to get a signed path for dev
• For prod:
  • Copy and customise:
    • main.tf.example to main.tf
    • terraform.tfvars.example to terraform.tfvars
  • make plan_prod to see changes
  • make provision_prod to apply changes
  • TO_SIGN="200x-100/https://beachape.com/images/octopress_with_container.png" make signature_for_prod to get a signed path