Noisy neighbours causing CPU entitlement usage of consistent load apps to increase

[ctlong]

Description

We were running an app using a consistent amount of CPU, which was under its CPU entitlement, and expected that the addition of a noisy neighbour (an app that uses a lot of CPU) to the system would not affect the original app. However, instead we saw that the original app increased the amount of CPU it was using.

Environment

• garden-runc-release version: ?
• Enabled configuration: ?
• IaaS: GCP
• Stemcell version: ?
• Kernel version: ?

Steps to reproduce

Here is the app we deployed that used a consistent amount of CPU:

package main

import (
        "crypto/rand"
        "crypto/rsa"
        "fmt"
        "net/http"
        "os"
)

func main() {
        http.HandleFunc("/", doWork)
        fmt.Println("listening...")
        err := http.ListenAndServe(":"+os.Getenv("PORT"), nil)
        if err != nil {
                panic(err)
        }
}

func doWork(res http.ResponseWriter, req *http.Request) {
        _, err := rsa.GenerateKey(rand.Reader, 1024)
        if err != nil {
                http.Error(res, err.Error(), http.StatusInternalServerError)
        }
}

We hit that app with a load-generating tool, e.g. hey. The command we used was: hey -q 2 -z 60m https://cpu-load-generator.example.com/. Scale the app as necessary to ensure that it remains under its entitlement.

Then we deployed dora and hit its stress_testers endpoint to create a noisy neighbour.

Once that's done you should see dora crushing the cores on your Diego Cell(/s). In addition, you should be able to observe that the CPU use of the original app increases.

Logs

None

Docker image (if relevant)

N/A

Cause

Not sure

Resolution

Not sure