references.md

Reference

• Custom Resource Definition
  • Metadata
    • name
  • Spec
    • owner
    • nodeSelector
    • namespaceQuota
    • namespacesMetadata
    • servicesMetadata
    • ingressClasses
    • ingressHostnames
    • storageClasses
    • containerRegistries
    • additionalRoleBindings
    • resourceQuotas
    • limitRanges
    • networkPolicies
    • externalServiceIPs
  • Status
    • size
    • namespaces
• Role Based Access Control
• Admission Controllers
• Command Options
• Created Resources

Custom Resource Definition

Capsule operator uses a single Custom Resources Definition (CRD) for Tenants. Please, see the Tenant Custom Resource Definition. In Caspule, Tenants are cluster wide resources. You need for cluster level permissions to work with tenants.

Metadata

name

Metadata name can contain any valid symbol from the regex: [a-z0-9]([-a-z0-9]*[a-z0-9])?.

Spec

owner

The field owner is the only mandatory spec in a Tenant manifest. It specifies the ownership of the tenant:

apiVersion: capsule.clastix.io/v1alpha1
kind: Tenant
metadata:
  name: tenant
spec:
  owner: # required
    name: <name>
    kind: <User|Group>

The user and group names should be valid identities. Capsule does not care about the authentication strategy used in the cluster and all the Kubernetes methods of Authentication are supported. The only requirement to use Capsule is to assign tenant users to the the group defined by --capsule-user-group option, which defaults to capsule.clastix.io.

Assignment to a group depends on the used authentication strategy.

For example, if you are using capsule.clastix.io, users authenticated through a X.509 certificate must have capsule.clastix.io as Organization: -subj "/CN=${USER}/O=capsule.clastix.io"

Users authenticated through an OIDC token must have

...
"users_groups": [
    "capsule.clastix.io",
    "other_group"
]

Permissions are controlled by RBAC.

nodeSelector

Field nodeSelector specifies the label to control the placement of pods on a given pool of worker nodes:

apiVersion: capsule.clastix.io/v1alpha1
kind: Tenant
metadata:
  name: tenant
spec:
  nodeSelector:
    <key>: <value>

All namesapces created within the tenant will have the annotation:

kind: Namespace
apiVersion: v1
metadata:
  annotations:
    scheduler.alpha.kubernetes.io/node-selector: 'key=value'

This annotation tells the Kubernetes scheduler to place pods on the nodes having that label:

kind: Pod
apiVersion: v1
metadata:
  name: sample
spec:
  nodeSelector:
    <key>: <value>

NB: While Capsule just enforces the annotation scheduler.alpha.kubernetes.io/node-selector at namespace level, the nodeSelector field in the pod template is under the control of the default PodNodeSelector enabled on the Kubernetes API server using the flag --enable-admission-plugins=PodNodeSelector.

Please, see how to Assigning Pods to Nodes documentation.

The tenant owner is not allowed to change or remove the annotation above from the namespace.

namespaceQuota

Field namespaceQuota specifies the maximum number of namespaces allowed for that tenant.

apiVersion: capsule.clastix.io/v1alpha1
kind: Tenant
metadata:
  name: tenant
spec:
  namespaceQuota: <quota>

Once the namespace quota assigned to the tenant has been reached, yhe tenant owner cannot create further namespaces.

namespacesMetadata

Field namespacesMetadata specifies additional labels and annotations the Capsule operator places on any Namespace in the tenant.

apiVersion: capsule.clastix.io/v1alpha1
kind: Tenant
metadata:
  name: tenant
spec:
  namespacesMetadata:
    additionalAnnotations:
      <annotations>
    additionalLabels:
      <key>: <value>

Al namespaces in the tenant will have:

kind: Namespace
apiVersion: v1
metadata:
  annotations:
    <annotations>
  labels:
    <key>: <value>

The tenant owner is not allowed to change or remove such labels and annotations from the namespace.

servicesMetadata

Field servicesMetadata specifies additional labels and annotations the Capsule operator places on any Service in the tenant.

apiVersion: capsule.clastix.io/v1alpha1
kind: Tenant
metadata:
  name: tenant
spec:
  servicesMetadata:
    additionalAnnotations:
      <annotations>
    additionalLabels:
      <key>: <value>

Al services in the tenant will have:

kind: Service
apiVersion: v1
metadata:
  annotations:
    <annotations>
  labels:
    <key>: <value>

The tenant owner is not allowed to change or remove such labels and annotations from the Service.

ingressClasses

Field ingressClasses specifies the IngressClass assigned to the tenant.

apiVersion: capsule.clastix.io/v1alpha1
kind: Tenant
metadata:
  name: tenant
spec:
  ingressClasses:
    allowed:
    - <class>
    allowedRegex: <regex>

Capsule assures that all the Ingress resources created in the tenant can use only one of the allowed IngressClass.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: <name>
  namespace:
  annotations:
    kubernetes.io/ingress.class: <class>

NB: Ingress resources are supported in both the versions, networking.k8s.io/v1beta1 and networking.k8s.io/v1.

Allowed IngressClasses are reported into namespaces as annotations, so the tenant owner can check them

kind: Namespace
apiVersion: v1
metadata:
  annotations:
    capsule.clastix.io/ingress-classes: <class>
    capsule.clastix.io/ingress-classes-regexp: <regex>

Any tentative of tenant owner to use a not allowed IngressClass will fail.

ingressHostnames

Field ingressHostnames specifies the allowed hostnames in Ingresses for the given tenant.

apiVersion: capsule.clastix.io/v1alpha1
kind: Tenant
metadata:
  name: tenant
spec:
  ingressHostnames:
     allowed:
     - <hostname>
     allowedRegex: <regex>

Capsule assures that all Ingress resources created in the tenant can use only one of the allowed hostnames.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: <name>
  namespace:
  annotations:
spec:
  rules:
  - host: <hostname>
    http: {}

NB: Ingress resources are supported in both the versions, networking.k8s.io/v1beta1 and networking.k8s.io/v1.

Any tentative of tenant owner to use one of not allowed hostnames will fail.

storageClasses

Field storageClasses specifies the StorageClasses assigned to the tenant.

apiVersion: capsule.clastix.io/v1alpha1
kind: Tenant
metadata:
  name: tenant
spec:
  storageClasses:
    allowed:
    - <class>
    allowedRegex: <regex>

Capsule assures that all PersistentVolumeClaim resources created in the tenant can use only one of the allowed StorageClasses.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: <name>
  namespace:
spec:
  storageClassName: <class>

Allowed StorageClasses are reported into namespaces as annotations, so the tenant owner can check them

kind: Namespace
apiVersion: v1
metadata:
  annotations:
    capsule.clastix.io/storage-classes: <class>
    capsule.clastix.io/storage-classes-regexp: <regex>

Any tentative of tenant owner to use a not allowed StorageClass will fail.

containerRegistries

Field containerRegistries specifies the ttrusted image registries assigned to the tenant.

apiVersion: capsule.clastix.io/v1alpha1
kind: Tenant
metadata:
  name: tenant
spec:
  containerRegistries:
     allowed:
     - <registry>
     allowedRegex: <regex>

Capsule assures that all Pods resources created in the tenant can use only one of the allowed trusted registries.

Allowed registries are reported into namespaces as annotations, so the tenant owner can check them

kind: Namespace
apiVersion: v1
metadata:
  annotations:
    capsule.clastix.io/allowed-registries-regexp: <regex>
    capsule.clastix.io/registries: <registry>

Any tentative of tenant owner to use a not allowed registry will fail.

NB: In case of naked and official images hosted on Docker Hub, Capsule is going to retrieve the registry even if it's not explicit: a busybox:latest Pod running on a Tenant allowing docker.io will not blocked, even if the image field is not explicit as docker.io/busybox:latest.

additionalRoleBindings

Field additionalRoleBindings specifies additional RoleBindings assigned to the tenant.

apiVersion: capsule.clastix.io/v1alpha1
kind: Tenant
metadata:
  name: tenant
spec:
  additionalRoleBindings:
  - clusterRoleName: <ClusterRole>
    subjects:
    - kind: <Group|User|ServiceAccount>
      apiGroup: rbac.authorization.k8s.io
      name: <name>

Capsule will ensure that all namespaces in the tenant always contain the RoleBinding for the given ClusterRole.

resourceQuotas

Field resourceQuotas specifies a list of ResourceQuota resources assigned to the tenant.

apiVersion: capsule.clastix.io/v1alpha1
kind: Tenant
metadata:
  name: tenant
spec:
  resourceQuotas:
  - hard:
      limits.cpu: <hard_value>
      limits.memory: <hard_value>
      requests.cpu: <hard_value>
      requests.memory: <hard_value>

Please, refer to ResourceQuota documentation for the subject.

The assigned quota are inherited by any namespace created in the tenant

kind: ResourceQuota
apiVersion: v1
metadata:
  name: compute
  namespace:
  labels:
    capsule.clastix.io/resource-quota=0
    capsule.clastix.io/tenant=tenant  
  annotations:
    # used resources in the tenant
    quota.capsule.clastix.io/used-limits.cpu=<tenant_used_value>       
    quota.capsule.clastix.io/used-limits.memory=<tenant_used_value>
    quota.capsule.clastix.io/used-requests.cpu=<tenant_used_value>
    quota.capsule.clastix.io/used-requests.memory=<tenant_used_value>
    # hard quota for the tenant
    quota.capsule.clastix.io/hard-limits.cpu=<tenant_hard_value>       
    quota.capsule.clastix.io/hard-limits.memory=<tenant_hard_value>
    quota.capsule.clastix.io/hard-requests.cpu=<tenant_hard_value>
    quota.capsule.clastix.io/hard-requests.memory=<tenant_hard_value>
spec:
  hard:
    limits.cpu: <hard_value>
    limits.memory: <hard_value>
    requests.cpu: <hard_value>
    requests.memory: <hard_value>
status:
  hard:
    limits.cpu: <namespace_hard_value>
    limits.memory: <namespace_hard_value>
    requests.cpu: <namespace_hard_value>
    requests.memory: <namespace_hard_value>
  used:
    limits.cpu: <namespace_used_value>
    limits.memory: <namespace_used_value>
    requests.cpu: <namespace_used_value>
    requests.memory: <namespace_used_value>

The Capsule operator aggregates ResourceQuota at tenant level, so that the hard quota is never crossed for the given tenant. This permits the tenant owner to consume resources in the tenant regardless of the namespace.

The annotations

quota.capsule.clastix.io/used-<resource>=<tenant_used_value>
quota.capsule.clastix.io/hard-<resource>=<tenant_hard_value>

are updated in realtime by Capsule, according to the actual aggredated usage of resource in the tenant.

NB: While Capsule controls quota at tenant level, at namespace level the quota enforcement is under the control of the default ResourceQuota Admission Controller enabled on the Kubernetes API server using the flag --enable-admission-plugins=ResourceQuota.

The tenant owner is not allowed to change or remove the ResourceQuota from the namespace.

limitRanges

Field limitRanges specifies the LimitRanges assigned to the tenant.

apiVersion: capsule.clastix.io/v1alpha1
kind: Tenant
metadata:
  name: tenant
spec:
  limitRanges:
  - limits:
    - type: Pod
      max:
        cpu: <value>
        memory: <value>
      min:
        cpu: <value>
        memory: <value> 
    - type: Container
      default:
        cpu: <value>
        memory: <value>
      defaultRequest:
        cpu: <value>
        memory: <value>
      max:
        cpu: <value>
        memory: <value>
      min:
        cpu: <value>
        memory: <value>          
    - type: PersistentVolumeClaim
      max:
        storage: <value>
      min:
        storage: <value>    

Please, refer to LimitRange documentation for the subject.

The assigned LimitRanges are inherited by any namespace created in the tenant

kind: LimitRange
apiVersion: v1
metadata:
  name: <name>
  namespace:
spec:
  limits:
  - type: Pod
    max:
      cpu: <value>
      memory: <value>
    min:
      cpu: <value>
      memory: <value> 
  - type: Container
    default:
      cpu: <value>
      memory: <value>
    defaultRequest:
      cpu: <value>
      memory: <value>
    max:
      cpu: <value>
      memory: <value>
    min:
      cpu: <value>
      memory: <value>          
  - type: PersistentVolumeClaim
    max:
      storage: <value>
    min:
      storage: <value>  

NB: Limit ranges enforcement for a single pod, container, and persistent volume claim is done by the default LimitRanger Admission Controller enabled on the Kubernetes API server: using the flag --enable-admission-plugins=LimitRanger.

Being the limit range specific of single resources, there is no aggregate to count.

The tenant owner is not allowed to change or remove LimitRanges from the namespace.

networkPolicies

Field networkPolicies specifies the NetworkPolicies assigned to the tenant.

apiVersion: capsule.clastix.io/v1alpha1
kind: Tenant
metadata:
  name: tenant
spec:
  networkPolicies:
  - policyTypes:
    - Ingress
    - Egress
    egress:
    - to:
      - ipBlock:
          cidr: <value>
    ingress:
    - from:
      - namespaceSelector: {}
      - podSelector: {}
      - ipBlock:
          cidr: <value>
    podSelector: {}

Please, refer to NetworkPolicies documentation for the subjects of a NetworkPolicy.

The assigned NetworkPolicies are inherited by any namespace created in the tenant.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: <name>
  namespace:
spec:
  podSelector: {}
  ingress:
  - from:
    - namespaceSelector: {}
    - podSelector: {}
    - ipBlock:
        cidr: <value>
  egress:
  - to:
    - ipBlock:
        cidr: <value>
  policyTypes:
  - Ingress
  - Egress

The tenant owner can create, patch and delete additional NetworkPolicy to refine the assigned one. However, the tenant owner cannot delete the NetworkPolicies set at tenant level.

externalServiceIPs

Field externalServiceIPs specifies the external IPs that can be used in Services with type ClusterIP.

apiVersion: capsule.clastix.io/v1alpha1
kind: Tenant
metadata:
  name: tenant
spec:
  externalServiceIPs:
    allowed:
    - <cidr>

Capsule will ensure that all Services in the tenant can contain only the allowed external IPs. This mitigate the [CVE-2020-8554] vulnerability where a potential attacker, able to create a Service with type ClusterIP and set the externalIPs field, can intercept traffic to that IP. Leave only the allowed CIDRs list to be set as externalIPs field in a Service with type ClusterIP.

To prevent users to set the externalIPs field, use an empty allowed list:

apiVersion: capsule.clastix.io/v1alpha1
kind: Tenant
metadata:
  name: tenant
spec:
  externalServiceIPs:
    allowed: []

NB: Missing of this controller, it exposes your cluster to the vulnerability [CVE-2020-8554].

Status

size

Status field size reports the number of namespaces belonging to the tenant. It is reported as NAMESPACE COUNT in the kubectl output:

$ kubectl get tnt
NAME      NAMESPACE QUOTA   NAMESPACE COUNT   OWNER NAME   OWNER KIND   NODE SELECTOR     AGE
cap       9                 1                 joe          User         {"pool":"cmp"}    5d4h
gas       6                 2                 alice        User         {"node":"worker"} 5d4h
oil       9                 3                 alice        User         {"pool":"cmp"}    5d4h
sample    9                 0                 alice        User         {"key":"value"}   29h

namespaces

Status field namespaces reports the list of all namespaces belonging to the tenant.

...
apiVersion: capsule.clastix.io/v1alpha1
kind: Tenant
metadata:
  name: tenant
spec:
...
status:
  namespaces: 
    oil-development
    oil-production
    oil-marketing
  size: 3

Role Based Access Control

In the current implementation, the Capsule operator requires cluster admin permissions to fully operate.

Admission Controllers

Capsule implements Kubernetes multi-tenancy capabilities using a minimum set of standard Admission Controllers enabled on the Kubernetes APIs server.

Here the list of required Admission Controllers you have to enable to get full support from Capsule:

• PodNodeSelector
• LimitRanger
• ResourceQuota
• MutatingAdmissionWebhook
• ValidatingAdmissionWebhook

In addition to the required controllers above, Capsule implements its own set through the Dynamic Admission Controller mechanism, providing callbacks to add further validation or resource patching.

To see Admission Controls installed by Capsule:

$ kubectl get ValidatingWebhookConfiguration
NAME                                       WEBHOOKS   AGE
capsule-validating-webhook-configuration   8          2h

$ kubectl get MutatingWebhookConfiguration
NAME                                       WEBHOOKS   AGE
capsule-mutating-webhook-configuration     1          2h

Command Options

The Capsule operator provides following command options:

Option	Description	Default
--metrics-addr	The address and port where /metrics are exposed.	127.0.0.1:8080
--enable-leader-election	Start a leader election client and gain leadership before executing the main loop.	true
--force-tenant-prefix	Force the tenant name as prefix for namespaces: <tenant_name>-<namespace>.	false
--zap-log-level	The log verbosity with a value from 1 to 10 or the basic keywords.	4
--zap-devel	The flag to get the stack traces for deep debugging.	null
--capsule-user-group	Override the Capsule group to which all tenant owners must belong.	capsule.clastix.io
--protected-namespace-regex	Disallows creation of namespaces matching the passed regexp.	null
--allow-ingress-hostname-collision	By default, Capsule allows Ingress hostname collision: set to false to enforce this policy.	true
--allow-tenant-ingress-hostnames-collision	Toggling this, Capsule will not check if a hostname collision is in place, allowing the creation of two or more Tenant resources although sharing the same allowed hostname(s).	false

Created Resources

Once installed, the Capsule operator creates the following resources in your cluster:

NAMESPACE       RESOURCE
                customresourcedefinition.apiextensions.k8s.io/tenants.capsule.clastix.io
                clusterrole.rbac.authorization.k8s.io/capsule-proxy-role
                clusterrole.rbac.authorization.k8s.io/capsule-metrics-reader
                mutatingwebhookconfiguration.admissionregistration.k8s.io/capsule-mutating-webhook-configuration
                validatingwebhookconfiguration.admissionregistration.k8s.io/capsule-validating-webhook-configuration
capsule-system  clusterrolebinding.rbac.authorization.k8s.io/capsule-manager-rolebinding
capsule-system  clusterrolebinding.rbac.authorization.k8s.io/capsule-proxy-rolebinding
capsule-system  secret/capsule-ca
capsule-system  secret/capsule-tls
capsule-system  service/capsule-controller-manager-metrics-service
capsule-system  service/capsule-webhook-service
capsule-system  deployment.apps/capsule-controller-manager