Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Tolerations to Build and BuildRun objects #1711

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Add Tolerations to Build and BuildRun objects #1711

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e963cdb
Add Tolerations to Build and BuildRun objects
dorzel Nov 13, 2024
7dc9025
Docs for Tolerations on Build and BuildRun objects
dorzel Nov 19, 2024
c35d173
Tests for Tolerations on Build and BuildRun objects
dorzel Nov 20, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 29 additions & 1 deletion .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,7 @@ jobs:
version: v0.24.0
node_image: kindest/node:${{ matrix.kubernetes }}
cluster_name: kind
config: test/kind/config.yaml
config: test/kind/config_three_node.yaml
wait: 120s
- name: Verify kind cluster
run: |
Expand All @@ -202,6 +202,34 @@ jobs:

exit 1
fi

kubectl wait --for=condition=Ready=true node/kind-worker --timeout=60s
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: create a for loop to iterate over the three nodes.

NODE_STATUS=$(kubectl get node kind-worker -o json | jq -r .'status.conditions[] | select(.type == "Ready") | .status')
if [ "${NODE_STATUS}" != "True" ]; then
echo "# Node is not ready:"
kubectl describe node kind-worker

echo "# Pods:"
kubectl get pod -A
echo "# Events:"
kubectl get events -A

exit 1
fi

kubectl wait --for=condition=Ready=true node/kind-worker2 --timeout=60s
NODE_STATUS=$(kubectl get node kind-worker2 -o json | jq -r .'status.conditions[] | select(.type == "Ready") | .status')
if [ "${NODE_STATUS}" != "True" ]; then
echo "# Node is not ready:"
kubectl describe node kind-worker2

echo "# Pods:"
kubectl get pod -A
echo "# Events:"
kubectl get events -A

exit 1
fi
- name: Install Tekton
env:
TEKTON_VERSION: ${{ matrix.tekton }}
Expand Down
117 changes: 117 additions & 0 deletions deploy/crds/shipwright.io_buildruns.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7535,6 +7535,45 @@ spec:
Build should take to execute.
format: duration
type: string
tolerations:
description: If specified, the pod's tolerations.
items:
description: |-
The pod this Toleration is attached to tolerates any taint that matches
the triple <key,value,effect> using the matching operator <operator>.
properties:
effect:
description: |-
Effect indicates the taint effect to match. Empty means match all taint effects.
When specified, allowed values are NoSchedule, PreferNoSchedule and NoExecute.
type: string
key:
description: |-
Key is the taint key that the toleration applies to. Empty means match all taint keys.
If the key is empty, operator must be Exists; this combination means to match all values and all keys.
type: string
operator:
description: |-
Operator represents a key's relationship to the value.
Valid operators are Exists and Equal. Defaults to Equal.
Exists is equivalent to wildcard for value, so that a pod can
tolerate all taints of a particular category.
type: string
tolerationSeconds:
description: |-
TolerationSeconds represents the period of time the toleration (which must be
of effect NoExecute, otherwise this field is ignored) tolerates the taint. By default,
it is not set, which means tolerate the taint forever (do not evict). Zero and
negative values will be treated as 0 (evict immediately) by the system.
format: int64
type: integer
value:
description: |-
Value is the taint value the toleration matches to.
If the operator is Exists, the value should be empty, otherwise just a regular string.
type: string
type: object
type: array
trigger:
description: Trigger defines the scenarios where a new build
should be triggered.
Expand Down Expand Up @@ -9753,6 +9792,45 @@ spec:
description: Timeout defines the maximum run time of this BuildRun.
format: duration
type: string
tolerations:
description: If specified, the pod's tolerations.
items:
description: |-
The pod this Toleration is attached to tolerates any taint that matches
the triple <key,value,effect> using the matching operator <operator>.
properties:
effect:
description: |-
Effect indicates the taint effect to match. Empty means match all taint effects.
When specified, allowed values are NoSchedule, PreferNoSchedule and NoExecute.
type: string
key:
description: |-
Key is the taint key that the toleration applies to. Empty means match all taint keys.
If the key is empty, operator must be Exists; this combination means to match all values and all keys.
type: string
operator:
description: |-
Operator represents a key's relationship to the value.
Valid operators are Exists and Equal. Defaults to Equal.
Exists is equivalent to wildcard for value, so that a pod can
tolerate all taints of a particular category.
type: string
tolerationSeconds:
description: |-
TolerationSeconds represents the period of time the toleration (which must be
of effect NoExecute, otherwise this field is ignored) tolerates the taint. By default,
it is not set, which means tolerate the taint forever (do not evict). Zero and
negative values will be treated as 0 (evict immediately) by the system.
format: int64
type: integer
value:
description: |-
Value is the taint value the toleration matches to.
If the operator is Exists, the value should be empty, otherwise just a regular string.
type: string
type: object
type: array
volumes:
description: |-
Volumes contains volume Overrides of the BuildStrategy volumes in case those are allowed
Expand Down Expand Up @@ -11959,6 +12037,45 @@ spec:
should take to execute.
format: duration
type: string
tolerations:
description: If specified, the pod's tolerations.
items:
description: |-
The pod this Toleration is attached to tolerates any taint that matches
the triple <key,value,effect> using the matching operator <operator>.
properties:
effect:
description: |-
Effect indicates the taint effect to match. Empty means match all taint effects.
When specified, allowed values are NoSchedule, PreferNoSchedule and NoExecute.
type: string
key:
description: |-
Key is the taint key that the toleration applies to. Empty means match all taint keys.
If the key is empty, operator must be Exists; this combination means to match all values and all keys.
type: string
operator:
description: |-
Operator represents a key's relationship to the value.
Valid operators are Exists and Equal. Defaults to Equal.
Exists is equivalent to wildcard for value, so that a pod can
tolerate all taints of a particular category.
type: string
tolerationSeconds:
description: |-
TolerationSeconds represents the period of time the toleration (which must be
of effect NoExecute, otherwise this field is ignored) tolerates the taint. By default,
it is not set, which means tolerate the taint forever (do not evict). Zero and
negative values will be treated as 0 (evict immediately) by the system.
format: int64
type: integer
value:
description: |-
Value is the taint value the toleration matches to.
If the operator is Exists, the value should be empty, otherwise just a regular string.
type: string
type: object
type: array
trigger:
description: Trigger defines the scenarios where a new build should
be triggered.
Expand Down
39 changes: 39 additions & 0 deletions deploy/crds/shipwright.io_builds.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2912,6 +2912,45 @@ spec:
should take to execute.
format: duration
type: string
tolerations:
description: If specified, the pod's tolerations.
items:
description: |-
The pod this Toleration is attached to tolerates any taint that matches
the triple <key,value,effect> using the matching operator <operator>.
properties:
effect:
description: |-
Effect indicates the taint effect to match. Empty means match all taint effects.
When specified, allowed values are NoSchedule, PreferNoSchedule and NoExecute.
type: string
key:
description: |-
Key is the taint key that the toleration applies to. Empty means match all taint keys.
If the key is empty, operator must be Exists; this combination means to match all values and all keys.
type: string
operator:
description: |-
Operator represents a key's relationship to the value.
Valid operators are Exists and Equal. Defaults to Equal.
Exists is equivalent to wildcard for value, so that a pod can
tolerate all taints of a particular category.
type: string
tolerationSeconds:
description: |-
TolerationSeconds represents the period of time the toleration (which must be
of effect NoExecute, otherwise this field is ignored) tolerates the taint. By default,
it is not set, which means tolerate the taint forever (do not evict). Zero and
negative values will be treated as 0 (evict immediately) by the system.
format: int64
type: integer
value:
description: |-
Value is the taint value the toleration matches to.
If the operator is Exists, the value should be empty, otherwise just a regular string.
type: string
type: object
type: array
trigger:
description: Trigger defines the scenarios where a new build should
be triggered.
Expand Down
6 changes: 5 additions & 1 deletion docs/build.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ A `Build` resource allows the user to define:
- retention
- volumes
- nodeSelector
- tolerations

A `Build` is available within a namespace.

Expand Down Expand Up @@ -94,6 +95,8 @@ To prevent users from triggering `BuildRun`s (_execution of a Build_) that will
| TriggerInvalidPipeline | Trigger type Pipeline is invalid. |
| OutputTimestampNotSupported | An unsupported output timestamp setting was used. |
| OutputTimestampNotValid | The output timestamp value is not valid. |
| NodeSelectorNotValid | The specified nodeSelector is not valid. |
| TolerationNotValid | The specified tolerations are not valid. |

## Configuring a Build

Expand Down Expand Up @@ -125,7 +128,8 @@ The `Build` definition supports the following fields:
- `spec.retention.ttlAfterSucceeded` - Specifies the duration for which a successful buildrun can exist.
- `spec.retention.failedLimit` - Specifies the number of failed buildrun that can exist.
- `spec.retention.succeededLimit` - Specifies the number of successful buildrun can exist.
- `spec.nodeSelector` - Specifies a selector which must match a node's labels for the build pod to be scheduled on that node.
- `spec.nodeSelector` - Specifies a selector which must match a node's labels for the build pod to be scheduled on that node. If nodeSelectors are specified in both a `Build` and `BuildRun`, `BuildRun` values take precedence.
- `spec.tolerations` - Specifies the tolerations for the build pod. Only `key`, `value`, and `operator` are supported. Only `NoSchedule` taint `effect` is supported. If tolerations are specified in both a `Build` and `BuildRun`, `BuildRun` values take precedence.

### Defining the Source

Expand Down
3 changes: 2 additions & 1 deletion docs/buildrun.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,8 @@ The `BuildRun` definition supports the following fields:
- `spec.output.timestamp` - Overrides the output timestamp configuration of the referenced build to instruct the build to change the output image creation timestamp to the specified value. When omitted, the respective build strategy tool defines the output image timestamp.
- `spec.output.vulnerabilityScan` - Overrides the output vulnerabilityScan configuration of the referenced build to run the vulnerability scan for the generated image.
- `spec.env` - Specifies additional environment variables that should be passed to the build container. Overrides any environment variables that are specified in the `Build` resource. The available variables depend on the tool used by the chosen build strategy.
- `spec.nodeSelector` - Specifies a selector which must match a node's labels for the build pod to be scheduled on that node.
- `spec.nodeSelector` - Specifies a selector which must match a node's labels for the build pod to be scheduled on that node. If nodeSelectors are specified in both a `Build` and `BuildRun`, `BuildRun` values take precedence.
- `spec.tolerations` - Specifies the tolerations for the build pod. Only `key`, `value`, and `operator` are supported. Only `NoSchedule` taint `effect` is supported. If tolerations are specified in both a `Build` and `BuildRun`, `BuildRun` values take precedence.

**Note**: The `spec.build.name` and `spec.build.spec` are mutually exclusive. Furthermore, the overrides for `timeout`, `paramValues`, `output`, and `env` can only be combined with `spec.build.name`, but **not** with `spec.build.spec`.

Expand Down
8 changes: 8 additions & 0 deletions pkg/apis/build/v1beta1/build_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,8 @@ const (
OutputTimestampNotValid BuildReason = "OutputTimestampNotValid"
// NodeSelectorNotValid indicates that the nodeSelector value is not valid
NodeSelectorNotValid BuildReason = "NodeSelectorNotValid"
// TolerationNotValid indicates that the Toleration value is not valid
TolerationNotValid BuildReason = "TolerationNotValid"

// AllValidationsSucceeded indicates a Build was successfully validated
AllValidationsSucceeded = "all validations succeeded"
Expand Down Expand Up @@ -183,6 +185,12 @@ type BuildSpec struct {
//
// +optional
NodeSelector map[string]string `json:"nodeSelector,omitempty"`

// If specified, the pod's tolerations.
// +optional
// +patchMergeKey=Key
// +patchStrategy=merge
Tolerations []corev1.Toleration `json:"tolerations,omitempty" patchStrategy:"merge" patchMergeKey:"Key"`
}

// BuildVolume is a volume that will be mounted in build pod during build step
Expand Down
6 changes: 6 additions & 0 deletions pkg/apis/build/v1beta1/buildrun_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,12 @@ type BuildRunSpec struct {
//
// +optional
NodeSelector map[string]string `json:"nodeSelector,omitempty"`

// If specified, the pod's tolerations.
// +optional
// +patchMergeKey=Key
// +patchStrategy=merge
Tolerations []corev1.Toleration `json:"tolerations,omitempty" patchStrategy:"merge" patchMergeKey:"Key"`
}

// BuildRunRequestedState defines the buildrun state the user can provide to override whatever is the current state.
Expand Down
1 change: 1 addition & 0 deletions pkg/reconciler/build/build.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ var validationTypes = [...]string{
validate.Envs,
validate.Triggers,
validate.NodeSelector,
validate.Tolerations,
}

// ReconcileBuild reconciles a Build object
Expand Down
18 changes: 17 additions & 1 deletion pkg/reconciler/build/build_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ import (
"k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/runtime/schema"
"k8s.io/apimachinery/pkg/types"
"k8s.io/apimachinery/pkg/util/validation"
"k8s.io/utils/ptr"
crc "sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
Expand Down Expand Up @@ -621,7 +622,22 @@ var _ = Describe("Reconcile Build", func() {
buildSample.Spec.NodeSelector = map[string]string{strings.Repeat("s", 64): "amd64"}
buildSample.Spec.Output.PushSecret = nil

statusCall := ctl.StubFunc(corev1.ConditionFalse, build.NodeSelectorNotValid, "name part must be no more than 63 characters")
statusCall := ctl.StubFunc(corev1.ConditionFalse, build.NodeSelectorNotValid, "name part "+validation.MaxLenError(63))
statusWriter.UpdateCalls(statusCall)

_, err := reconciler.Reconcile(context.TODO(), request)
Expect(err).To(BeNil())
Expect(statusWriter.UpdateCallCount()).To(Equal(1))
})
})

Context("when Tolerations is specified", func() {
It("should fail to validate when the Toleration is invalid", func() {
// set Toleration to be invalid
buildSample.Spec.Tolerations = []corev1.Toleration{{Key: strings.Repeat("s", 64), Operator: "Equal", Value: "test-value"}}
buildSample.Spec.Output.PushSecret = nil

statusCall := ctl.StubFunc(corev1.ConditionFalse, build.TolerationNotValid, "name part "+validation.MaxLenError(63))
statusWriter.UpdateCalls(statusCall)

_, err := reconciler.Reconcile(context.TODO(), request)
Expand Down
1 change: 1 addition & 0 deletions pkg/reconciler/buildrun/buildrun.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,7 @@ func (r *ReconcileBuildRun) Reconcile(ctx context.Context, request reconcile.Req
validate.NewBuildName(build),
validate.NewEnv(build),
validate.NewNodeSelector(build),
validate.NewTolerations(build),
)

// an internal/technical error during validation happened
Expand Down
Loading
Loading