If you are using a released version of Kubernetes, you should refer to the docs that go with that version.
The latest release of this document can be found [here](http://releases.k8s.io/release-1.4/docs/proposals/volume-provisioning.md).Documentation for other releases can be found at releases.k8s.io.
Real Kubernetes clusters have a variety of volumes which differ widely in size, iops performance, retention policy, and other characteristics. Administrators need a way to dynamically provision volumes of these different types to automatically meet user demand.
A new mechanism called 'storage classes' is proposed to provide this capability.
In Kubernetes 1.2, an alpha form of limited dynamic provisioning was added that allows a single volume type to be provisioned in clouds that offer special volume types.
In Kubernetes 1.3, a label selector was added to persistent volume claims to allow administrators to create a taxonomy of volumes based on the characteristics important to them, and to allow users to make claims on those volumes based on those characteristics. This allows flexibility when claiming existing volumes; the same flexibility is needed when dynamically provisioning volumes.
After gaining experience with dynamic provisioning after the 1.2 release, we want to create a more flexible feature that allows configuration of how different storage classes are provisioned and supports provisioning multiple types of volumes within a single cloud.
One of our goals is to enable administrators to create out-of-tree provisioners, that is, provisioners whose code does not live in the Kubernetes project. Our experience since the 1.2 release with dynamic provisioning has shown that it is impossible to anticipate every aspect and manner of provisioning that administrators will want to perform. The proposed design should not prevent future work to allow out-of-tree provisioners.
This design represents the minimally viable changes required to provision based on storage classe configuration. Additional incremental features may be added as a separte effort.
We propose that:
-
For the base impelementation storage class and volume selectors are mutually exclusive.
-
An api object will be incubated in storage.k8s.io/v1beta1 to hold the a
StorageClass
API resource. Each StorageClass object contains parameters required by the provisioner to provision volumes of that class. These parameters are opaque to the user. -
PersistentVolume.Spec.Class
attribute is added to volumes. This attribute is optional and specifies whichStorageClass
instance represents storage characteristics of a particular PV.During incubation,
Class
is an annotation and not actual attribute. -
PersistentVolume
instances do not require labels by the provisioner. -
PersistentVolumeClaim.Spec.Class
attribute is added to claims. This attribute specifies that only a volume with equalPersistentVolume.Spec.Class
value can satisfy a claim.During incubation,
Class
is just an annotation and not actual attribute. -
The existing provisioner plugin implementations be modified to accept parameters as specified via
StorageClass
. -
The persistent volume controller modified to invoke provisioners using
StorageClass
configuration and bind claims withPersistentVolumeClaim.Spec.Class
to volumes with equivilantPersistentVolume.Spec.Class
-
The existing alpha dynamic provisioning feature be phased out in the next release.
-
When a new claim is submitted, the controller attempts to find an existing volume that will fulfill the claim.
-
If the claim has non-empty
claim.Spec.Class
, only PVs with the samepv.Spec.Class
are considered. -
If the claim has empty
claim.Spec.Class
, all existing PVs are considered.
All "considered" volumes are evaluated and the smallest matching volume is bound to the claim.
-
-
If no volume is found for the claim and
claim.Spec.Class
is not set or is empty string dynamic provisioning is disabled. -
If
claim.Spec.Class
is set the controller tries to find instance of StorageClass with this name. If no such StorageClass is found, the controller goes back to step 1. and periodically retries finding a matching volume or storage class again until a match is found. The claim isPending
during this period. -
With StorageClass instance, the controller finds volume plugin specified by StorageClass.Provisioner.
-
All provisioners are in-tree; they implement an interface called
ProvisionableVolumePlugin
, which has a method calledNewProvisioner
that returns a new provisioner. -
The controller calls volume plugin
Provision
with Parameters from theStorageClass
configuration object. -
If
Provision
returns an error, the controller generates an event on the claim and goes back to step 1., i.e. it will retry provisioning periodically -
If
Provision
returns no error, the controller creates the returnedapi.PersistentVolume
, fills itsClass
attribute withclaim.Spec.Class
and makes it already bound to the claim -
If the create operation for the
api.PersistentVolume
fails, it is retried -
If the create operation does not succeed in reasonable time, the controller attempts to delete the provisioned volume and creates an event on the claim
Existing behavior is un-changed for claims that do not specify claim.Spec.Class
.
A new API group should hold the API for storage classes, following the pattern
of autoscaling, metrics, etc. To allow for future storage-related APIs, we
should call this new API group storage.k8s.io
and incubate in storage.k8s.io/v1beta1.
Storage classes will be represented by an API object called StorageClass
:
package storage
// StorageClass describes the parameters for a class of storage for
// which PersistentVolumes can be dynamically provisioned.
//
// StorageClasses are non-namespaced; the name of the storage class
// according to etcd is in ObjectMeta.Name.
type StorageClass struct {
unversioned.TypeMeta `json:",inline"`
ObjectMeta `json:"metadata,omitempty"`
// Provisioner indicates the type of the provisioner.
Provisioner string `json:"provisioner,omitempty"`
// Parameters for dynamic volume provisioner.
Parameters map[string]string `json:"parameters,omitempty"`
}
PersistentVolumeClaimSpec
and PersistentVolumeSpec
both get Class attribute
(the existing annotation is used during incubation):
type PersistentVolumeClaimSpec struct {
// Name of requested storage class. If non-empty, only PVs with this
// pv.Spec.Class will be considered for binding and if no such PV is
// available, StorageClass with this name will be used to dynamically
// provision the volume.
Class string
...
}
type PersistentVolumeSpec struct {
// Name of StorageClass instance that this volume belongs to.
Class string
...
}
Storage classes are natural to think of as a global resource, since they:
- Align with PersistentVolumes, which are a global resource
- Are administrator controlled
With the scheme outlined above the provisioner creates PVs using parameters specified in the StorageClass
object.
struct volume.VolumeOptions
(containing parameters for a provisioner plugin)
will be extended to contain StorageClass.Parameters.
The existing provisioner implementations will be modified to accept the StorageClass configuration object.
The persistent volume controller will be modified to implement the new
workflow described in this proposal. The changes will be limited to the
provisionClaimOperation
method, which is responsible for invoking the
provisioner and to favor existing volumes before provisioning a new one.
This example shows two storage classes, "aws-fast" and "aws-slow".
apiVersion: v1
kind: StorageClass
metadata:
name: aws-fast
provisioner: kubernetes.io/aws-ebs
parameters:
zone: us-east-1b
type: ssd
apiVersion: v1
kind: StorageClass
metadata:
name: aws-slow
provisioner: kubernetes.io/aws-ebs
parameters:
zone: us-east-1b
type: spinning
-
Annotation
volume.alpha.kubernetes.io/storage-class
is used instead ofclaim.Spec.Class
andvolume.Spec.Class
during incubation. -
claim.Spec.Selector
andclaim.Spec.Class
are mutually exclusive. User can either match existing volumes withSelector
XOR match existing volumes withClass
and get dynamic provisioning by usingClass
. This simplifies initial PR and also provisioners.
Since the volume.alpha.kubernetes.io/storage-class
is in use a StorageClass
must be defined to support provisioning. No default is assumed as before.