Resource limits for step containers

[lonix1]

I originally posted on discord, twice, without resolution. I assume that means what I've described isn't currently possible, hence this feature request.

Clear and concise description of the problem

In a memory constrained VPS, I'd like to set memory and cpu limits, because (even for a small build) the memory use spikes and impacts other docker services. I think the issue is not for the server or agent containers, but rather the ephemeral "step containers".

Typically one would set (in docker-compose.yml) resource limits for such containers:

services:
  foo:
    # ...
    deploy:
      resources:
        limits:
          cpus: '0.25'
          memory: 150m
    memswap_limit: 1g

I found env vars in the docs but evidently they don't represent the same idea as the code above.

services:
  woodpecker:
    # ...
    environment:
      - WOODPECKER_LIMIT_MEM=157286400          # 150MiB: only a number works, suffixes are not allowed (e.g. 'm", 'mb', 'M', 'Mi', 'MiB'), which is fine
      - WOODPECKER_LIMIT_MEM_SWAP=1073741824    # 1GiB
      - WOODPECKER_LIMIT_CPU_QUOTA=             # ?
      - WOODPECKER_LIMIT_CPU_SHARES=            # ?

The docs state:

WOODPECKER_LIMIT_CPU_QUOTA: "The number of microseconds per CPU period that the container is limited to before throttled. There is no limit if 0."

and

WOODPECKER_LIMIT_CPU_SHARES: "The relative weight vs. other containers."

I'm unsure why these options were made available, or what is their use case - my guess is woodpecker is using a go docker library which exposes this functionality, and so wraps it?

Suggested solution

What's missing is cpu throttling by percentage, like docker does it. So for example:

- WOODPECKER_LIMIT_CPU_PERCENT=0.25

Alternative

There is no alternative, from what I can tell? The step containers use as much CPU as they need.

Additional context

Thanks.

Validations

• Checked that the feature isn't part of the next version already [https://woodpecker-ci.org/faq#which-version-of-woodpecker-should-i-use]
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

Environment

1.0.0
docker

[lonix1]

The above was in expectation of a problem .But now I actually ran into the problem for real.

(It took a LONG time to diagnose the issue - the problem needs to be "surfaced" better. I had to enable trace logging and watch woodpecker server container logs.)

I set memory and cpu constraints for the server and agent containers, but it's unclear how to do that for step containers - as discussed above.

On a "smallish" VPS, a step container gobbled up all available memory and the pipeline was cancelled. Only after looking into the trace logs did I find the reason:

{"level":"trace","repo_id":"80","pipeline_id":"346","workflow_id":"345","time":"2023-08-26T04:45:02+01:00","caller":"/woodpecker/src/github.com/woodpecker-ci/woodpecker/server/grpc/rpc.go:261","message":"gRPC Done with state: rpc.State{Step:\"\", Exited:true, ExitCode:1, Started:1693017761, Finished:1693017902, Error:\"wp_01h8qvht4q0yyanqgx81vrsqaf_0_step_3 : received oom kill\"}"}

received oom kill

I would expect that in such a case the system would use swap memory instead, but it didn't.

We need to be able to set resource constraints on step containers to avoid this issue. And it should be logged somewhere that is easily accessible so the woodpecker admin knows to tweak the settings, or buy more memory.

[lonix1]

Related: #447