admin-quota-overview.adoc

Resources managed by quota

A resource quota, defined by a ResourceQuota object, provides constraints that limit aggregate resource consumption per project. It can limit the quantity of objects that can be created in a project by type, as well as the total amount of compute resources and storage that may be consumed by resources in that project.

The following describes the set of compute resources and object types that may be managed by a quota.

Note	A pod is in a terminal state if status.phase is Failed or Succeeded.

Table 1. Compute resources managed by quota

Resource Name	Description
cpu	The sum of CPU requests across all pods in a non-terminal state cannot exceed this value. cpu and requests.cpu are the same value and can be used interchangeably.
memory	The sum of memory requests across all pods in a non-terminal state cannot exceed this value. memory and requests.memory are the same value and can be used interchangeably.
ephemeral-storage	The sum of local ephemeral storage requests across all pods in a non-terminal state cannot exceed this value. ephemeral-storage and requests.ephemeral-storage are the same value and can be used interchangeably. This resource is available only if you enabled the ephemeral storage technology preview. This feature is disabled by default.
requests.cpu	The sum of CPU requests across all pods in a non-terminal state cannot exceed this value. cpu and requests.cpu are the same value and can be used interchangeably.
requests.memory	The sum of memory requests across all pods in a non-terminal state cannot exceed this value. memory and requests.memory are the same value and can be used interchangeably.
requests.ephemeral-storage	The sum of ephemeral storage requests across all pods in a non-terminal state cannot exceed this value. ephemeral-storage and requests.ephemeral-storage are the same value and can be used interchangeably. This resource is available only if you enabled the ephemeral storage technology preview. This feature is disabled by default.
limits.cpu	The sum of CPU limits across all pods in a non-terminal state cannot exceed this value.
limits.memory	The sum of memory limits across all pods in a non-terminal state cannot exceed this value.
limits.ephemeral-storage	The sum of ephemeral storage limits across all pods in a non-terminal state cannot exceed this value. This resource is available only if you enabled the ephemeral storage technology preview. This feature is disabled by default.

Table 2. Storage resources managed by quota

Resource Name	Description
requests.storage	The sum of storage requests across all persistent volume claims in any state cannot exceed this value.
persistentvolumeclaims	The total number of persistent volume claims that can exist in the project.
<storage-class-name>.storageclass.storage.k8s.io/requests.storage	The sum of storage requests across all persistent volume claims in any state that have a matching storage class, cannot exceed this value.
<storage-class-name>.storageclass.storage.k8s.io/persistentvolumeclaims	The total number of persistent volume claims with a matching storage class that can exist in the project.

Table 3. Object counts managed by quota

Resource Name	Description
pods	The total number of pods in a non-terminal state that can exist in the project.
replicationcontrollers	The total number of replication controllers that can exist in the project.
resourcequotas	The total number of resource quotas that can exist in the project.
services	The total number of services that can exist in the project.
secrets	The total number of secrets that can exist in the project.
configmaps	The total number of ConfigMap objects that can exist in the project.
persistentvolumeclaims	The total number of persistent volume claims that can exist in the project.
openshift.io/imagestreams	The total number of image streams that can exist in the project.

You can configure an object count quota for these standard namespaced resource types using the count/<resource>.<group> syntax.

$ oc create quota <name> --hard=count/<resource>.<group>=<quota> (1)

1. <resource> is the name of the resource, and <group> is the API group, if applicable. Use the kubectl api-resources command for a list of resources and their associated API groups.

Setting resource quota for extended resources

Overcommitment of resources is not allowed for extended resources, so you must specify requests and limits for the same extended resource in a quota. Currently, only quota items with the prefix requests. are allowed for extended resources. The following is an example scenario of how to set resource quota for the GPU resource nvidia.com/gpu.

Procedure

1. To determine how many GPUs are available on a node in your cluster, use the following command:

   $ oc describe node ip-172-31-27-209.us-west-2.compute.internal | egrep 'Capacity|Allocatable|gpu'

   Example output

                       openshift.com/gpu-accelerator=true
   Capacity:
    nvidia.com/gpu:  2
   Allocatable:
    nvidia.com/gpu:  2
    nvidia.com/gpu:  0           0

   In this example, 2 GPUs are available.

2. Use this command to set a quota in the namespace nvidia. In this example, the quota is 1:

   $ cat gpu-quota.yaml

   Example output

   apiVersion: v1
   kind: ResourceQuota
   metadata:
     name: gpu-quota
     namespace: nvidia
   spec:
     hard:
       requests.nvidia.com/gpu: 1

3. Create the quota with the following command:

   $ oc create -f gpu-quota.yaml

   Example output

   resourcequota/gpu-quota created

4. Verify that the namespace has the correct quota set using the following command:

   $ oc describe quota gpu-quota -n nvidia

   Example output

   Name:                    gpu-quota
   Namespace:               nvidia
   Resource                 Used  Hard
   --------                 ----  ----
   requests.nvidia.com/gpu  0     1

5. Run a pod that asks for a single GPU with the following command:

   $ oc create pod gpu-pod.yaml

   Example output

   apiVersion: v1
   kind: Pod
   metadata:
     generateName: gpu-pod-s46h7
     namespace: nvidia
   spec:
     restartPolicy: OnFailure
     containers:
     - name: rhel7-gpu-pod
       image: rhel7
       env:
         - name: NVIDIA_VISIBLE_DEVICES
           value: all
         - name: NVIDIA_DRIVER_CAPABILITIES
           value: "compute,utility"
         - name: NVIDIA_REQUIRE_CUDA
           value: "cuda>=5.0"
   
       command: ["sleep"]
       args: ["infinity"]
   
       resources:
         limits:
           nvidia.com/gpu: 1

6. Verify that the pod is running bwith the following command:

   $ oc get pods

   Example output

   NAME              READY     STATUS      RESTARTS   AGE
   gpu-pod-s46h7     1/1       Running     0          1m

7. Verify that the quota Used counter is correct by running the following command:

   $ oc describe quota gpu-quota -n nvidia

   Example output

   Name:                    gpu-quota
   Namespace:               nvidia
   Resource                 Used  Hard
   --------                 ----  ----
   requests.nvidia.com/gpu  1     1

8. Using the following command, attempt to create a second GPU pod in the nvidia namespace. This is technically available on the node because it has 2 GPUs:

   $ oc create -f gpu-pod.yaml

   Example output

   Error from server (Forbidden): error when creating "gpu-pod.yaml": pods "gpu-pod-f7z2w" is forbidden: exceeded quota: gpu-quota, requested: requests.nvidia.com/gpu=1, used: requests.nvidia.com/gpu=1, limited: requests.nvidia.com/gpu=1

   This Forbidden error message occurs because you have a quota of 1 GPU and this pod tried to allocate a second GPU, which exceeds its quota.

Quota scopes

Each quota can have an associated set of scopes. A quota only measures usage for a resource if it matches the intersection of enumerated scopes.

Adding a scope to a quota restricts the set of resources to which that quota can apply. Specifying a resource outside of the allowed set results in a validation error.

Scope	Description
Terminating	Match pods where spec.activeDeadlineSeconds >= 0.
NotTerminating	Match pods where spec.activeDeadlineSeconds is nil.
BestEffort	Match pods that have best effort quality of service for either cpu or memory.
otBestEffort	Match pods that do not have best effort quality of service for cpu and memory.

A BestEffort scope restricts a quota to limiting the following resources:

• pods

A Terminating, NotTerminating, and NotBestEffort scope restricts a quota to tracking the following resources:

• pods

• memory

• requests.memory

• limits.memory

• cpu

• requests.cpu

• limits.cpu

• ephemeral-storage

• requests.ephemeral-storage

• limits.ephemeral-storage

Note	Ephemeral storage requests and limits apply only if you enabled the ephemeral storage technology preview. This feature is disabled by default.