Skip to content

Commit

Permalink
ci: Rewrite e2e test suite from bash to golang (#217)
Browse files Browse the repository at this point in the history
* ci: Implement first draft of new e2e test suite

* ci: Ensure that e2e are not cachable

* fix(ci): Adjust paths

* test: Make cleanup more robust (remove finalizer on CR)

* fix: Linting issue and run e2e without waiting for olm

* test: Improve error handling

* ci(test): Refactor e2e initialization

* test: Extend initial install, reorder e2e execution

* ci: Fixup script path

* test: Use sdk to change scc instead of oc cli

* fixup

* test: Refactor install test

* test: Check agent logs for successful connection

* test: Wait for controller-manager before applying CR, Readme update

* test: Use correct agent download key secret

* test: Implement update install, introduce test api

* test: Add test to check reconciliation of new operator

* test: Refactor config and logging

* test: Cleanup test API

* test: Add initial install test, reuse logic if possible

* test: Add multi-backend test to new e2e framework

* test: Exec into pod to check configuration

* test: Remove old e2e test bash script

* test: Add explicit test for keysSecret and inline secret

* chore: Fix linting issues

* chore: Add support for building with podman

* fix: Restore Dockerfile to work on non-amd64 hosts

* chore: Add support for local podman build via buildkit

If running on podman, the Dockerfile syntax is not accepted in all cases
and the build might behave differently compared to Docker. Buildkit is
the foundation of the docker build logic and can be used in isolation,
also as part of a running container. Therefore, the Makefile now detects
podman on docker on demand and installs the buildctl command line if
necessary. The build is always executed via buildctl now, which should
provide a consistent build behavior.

* test: Align test suite with k8s multi-backend changes
  • Loading branch information
konrad-ohms authored Oct 7, 2024
1 parent 846d2e7 commit c8e1573
Show file tree
Hide file tree
Showing 16 changed files with 1,019 additions and 378 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -63,3 +63,4 @@ instana-agent-operator
# CI
backend.cfg

e2e/.env
4 changes: 2 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#
# (c) Copyright IBM Corp. 2021
# (c) Copyright IBM Corp. 2021, 2024
# (c) Copyright Instana Inc.
#

Expand Down Expand Up @@ -75,4 +75,4 @@ RUN chown -R ${USER_UID}:${USER_UID} .cache
RUN chmod -R 777 .cache

USER ${USER_UID}:${USER_UID}
ENTRYPOINT ["/manager"]
ENTRYPOINT ["/manager"]
45 changes: 38 additions & 7 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,12 @@ ifeq ($(uname), Darwin)
get_ip_addr := ipconfig getifaddr en0
endif

# Detect if podman or docker is available locally
ifeq ($(shell command -v podman 2> /dev/null),)
CONTAINER_CMD = docker
else
CONTAINER_CMD = podman
endif

all: build

Expand Down Expand Up @@ -88,13 +94,16 @@ vet: ## Run go vet against code
lint: golangci-lint ## Run the golang-ci linter
$(GOLANGCI_LINT) run --timeout 5m

EXCLUDED_TEST_DIRS = mocks
EXCLUDED_TEST_DIRS = mocks e2e
EXCLUDE_PATTERN = $(shell echo $(EXCLUDED_TEST_DIRS) | sed 's/ /|/g')
PACKAGES = $(shell go list ./... | grep -vE "$(EXCLUDE_PATTERN)" | tr '\n' ' ')
KUBEBUILDER_ASSETS=$(shell $(ENVTEST) use $(ENVTEST_K8S_VERSION) -p path)
test: gen-mocks manifests generate fmt vet lint envtest ## Run tests but ignore specific directories that match EXCLUDED_TEST_DIRS
KUBEBUILDER_ASSETS="$(KUBEBUILDER_ASSETS)" go test $(PACKAGES) -coverprofile=coverage.out

.PHONY: e2e
e2e:
go test -timeout=10m -count=1 -v github.com/instana/instana-agent-operator/e2e

##@ Build

Expand All @@ -105,12 +114,13 @@ run: export DEBUG_MODE=true
run: generate fmt vet manifests ## Run against the configured Kubernetes cluster in ~/.kube/config (run the "install" target to install CRDs into the cluster)
go run ./

docker-build: test ## Build docker image with the manager.
docker build --build-arg VERSION=${VERSION} --build-arg GIT_COMMIT=${GIT_COMMIT} --build-arg DATE="$$(date)" -t ${IMG} .
docker-build: test container-build ## Build docker image with the manager.

docker-push: ## Push the docker image with the manager.
docker push ${IMG}
${CONTAINER_CMD} push ${IMG}

container-build: buildctl
$(BUILDCTL) --addr=${CONTAINER_CMD}-container://buildkitd build --frontend=dockerfile.v0 --local context=. --local dockerfile=. --output type=oci,name=${IMG} --opt build-arg:VERSION=0.0.1 --opt build-arg:GIT_COMMIT=${GIT_COMMIT} --opt build-arg:DATE="$$(date)" | $(CONTAINER_CMD) load

##@ Deployment

Expand Down Expand Up @@ -167,6 +177,26 @@ endif
operator-sdk: ## Download the Operator SDK binary locally if necessary.
$(call curl-get-tool,$(OPERATOR_SDK),https://github.com/operator-framework/operator-sdk/releases/download/v1.16.0,operator-sdk_$${OS}_$${ARCH})

BUILDCTL = $(shell pwd)/bin/buildctl
BUILDKITD_CONTAINER_NAME = buildkitd
# Test if buildctl is available in the GOPATH, if not, set to local and download if needed
buildctl: ## Download the buildctl cli locally if necessary.
@if [ "`podman ps -a -q -f name=$(BUILDKITD_CONTAINER_NAME)`" ]; then \
if [ "`podman ps -aq -f status=exited -f name=$(BUILDKITD_CONTAINER_NAME)`" ]; then \
echo "Starting buildkitd container $(BUILDKITD_CONTAINER_NAME)"; \
$(CONTAINER_CMD) start $(BUILDKITD_CONTAINER_NAME) || true; \
echo "Allowing 5 seconds to bootup"; \
sleep 5; \
else \
echo "Buildkit daemon is already running, skip container creation"; \
fi \
else \
echo "$(BUILDKITD_CONTAINER_NAME) container is not present, launching it now"; \
$(CONTAINER_CMD) run -d --name buildkitd --privileged docker.io/moby/buildkit:v0.16.0; \
echo "Allowing 5 seconds to bootup"; \
sleep 5; \
fi
$(call go-install-tool,$(BUILDCTL),github.com/moby/buildkit/cmd/[email protected])

# go-install-tool will 'go get' any package $2 and install it to $1.
PROJECT_DIR := $(shell dirname $(abspath $(lastword $(MAKEFILE_LIST))))
Expand Down Expand Up @@ -220,8 +250,9 @@ bundle: operator-sdk manifests kustomize ## Create the OLM bundle
$(OPERATOR_SDK) bundle validate ./bundle

.PHONY: bundle-build
bundle-build: ## Build the bundle image for OLM.
docker build -f bundle.Dockerfile -t $(BUNDLE_IMG) .
bundle-build: buildctl ## Build the bundle image for OLM.
#docker build -f bundle.Dockerfile -t $(BUNDLE_IMG) .
$(BUILDCTL) --addr=${CONTAINER_CMD}-container://buildkitd build --frontend gateway.v0 --opt source=docker/dockerfile --opt filename=./bundle.Dockerfile --local context=. --local dockerfile=. --output type=oci,name=${BUNDLE_IMG} | $(CONTAINER_CMD) load

controller-yaml: manifests kustomize ## Output the YAML for deployment, so it can be packaged with the release. Use `make --silent` to suppress other output.
cd config/manager && $(KUSTOMIZE) edit set image "instana/instana-agent-operator=$(IMG)"
Expand All @@ -243,4 +274,4 @@ gen-mocks: get-mockgen
mockgen --source ./pkg/k8s/object/builders/common/builder/builder.go --destination ./mocks/builder_mock.go --package mocks
mockgen --source ./pkg/json_or_die/json.go --destination ./mocks/json_or_die_marshaler_mock.go --package mocks
mockgen --source ./pkg/k8s/operator/status/agent_status_manager.go --destination ./mocks/agent_status_manager_mock.go --package mocks
mockgen --source ./pkg/k8s/operator/lifecycle/dependent_lifecycle_manager.go --destination ./mocks/dependent_lifecycle_manager_mock.go --package mocks
mockgen --source ./pkg/k8s/operator/lifecycle/dependent_lifecycle_manager.go --destination ./mocks/dependent_lifecycle_manager_mock.go --package mocks
29 changes: 29 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,3 +131,32 @@ Now you should have a successful running Operator.
To remove the Operator again, run:
* `kubectl delete -f config/samples/instana_v1_instanaagent_demo.yaml`
* `make undeploy`.

### Running tests

Unit tests can be executed by running `make test` without adjustments of the local environment.

For end-to-end testing, it is necessary to have a valid kubeconfig in the default location and to export variables before starting the test.
An example template file is available in [e2e/.env.template](./e2e/.env.template), copy it to `./e2e/.env` and adjust it accordingly.

The test can be executed by sourcing the config `source ./e2e/.env` and running `make e2e` or by using the VSCode.

For VSCode, ensure to have a valid `.vscode/settings.json` in your root folder.

Example:

```json
{
"wcaForGP.enable": true,
"go.testEnvVars": {
"KUBEBUILDER_ASSETS": "~/.local/share/kubebuilder-envtest/k8s/1.30.0-linux-amd64",
"INSTANA_API_KEY": "xxx",
"ARTIFACTORY_USERNAME": "xxx",
"ARTIFACTORY_PASSWORD": "xxx",
"OPERATOR_IMAGE_NAME": "xxx",
"OPERATOR_IMAGE_TAG": "xxx"
},
"wca.enable": false,
"go.testTimeout": "600s"
}
```
22 changes: 17 additions & 5 deletions ci/pipeline.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -524,13 +524,19 @@ jobs:
INSTANA_ENDPOINT_HOST: ((instana-qa.endpoint_host))
INSTANA_ENDPOINT_PORT: 443
BUILD_BRANCH: main
INSTANA_API_KEY: ((qa-instana-api-token))
INSTANA_API_KEY: ((qa-instana-agent-key))
ARTIFACTORY_USERNAME: ((delivery-instana-io-internal-project-artifact-read-writer-creds.username))
ARTIFACTORY_PASSWORD: ((delivery-instana-io-internal-project-artifact-read-writer-creds.password))
inputs:
- name: pipeline-source
run:
path: pipeline-source/ci/scripts/end-to-end-test.sh
path: bash
args:
- -ceu
- |
cd pipeline-source
bash ./ci/scripts/cluster-authentication.sh
make e2e
on_success:
put: gh-status
inputs: [ pipeline-source ]
Expand Down Expand Up @@ -603,13 +609,19 @@ jobs:
INSTANA_API_URL: ((instana-qa.api_url))
INSTANA_API_TOKEN: ((instana-qa.api_token))
BUILD_BRANCH: main
INSTANA_API_KEY: ((qa-instana-api-token))
INSTANA_API_KEY: ((qa-instana-agent-key))
ARTIFACTORY_USERNAME: ((delivery-instana-io-internal-project-artifact-read-writer-creds.username))
ARTIFACTORY_PASSWORD: ((delivery-instana-io-internal-project-artifact-read-writer-creds.password))
inputs:
- name: pipeline-source
run:
path: pipeline-source/ci/scripts/end-to-end-test.sh
path: bash
args:
- -ceu
- |
cd pipeline-source
bash ./ci/scripts/cluster-authentication.sh
make e2e
on_success:
put: gh-status
inputs: [ pipeline-source ]
Expand Down Expand Up @@ -684,7 +696,7 @@ jobs:
# INSTANA_API_URL: ((instana-qa.api_url))
# INSTANA_API_TOKEN: ((instana-qa.api_token))
# BUILD_BRANCH: main
# INSTANA_API_KEY: ((qa-instana-api-token))
# INSTANA_API_KEY: ((qa-instana-agent-key))
# ARTIFACTORY_USERNAME: ((delivery-instana-io-internal-project-artifact-read-writer-creds.username))
# ARTIFACTORY_PASSWORD: ((delivery-instana-io-internal-project-artifact-read-writer-creds.password))
# inputs:
Expand Down
24 changes: 18 additions & 6 deletions ci/pr-pipeline.yml
Original file line number Diff line number Diff line change
Expand Up @@ -565,7 +565,7 @@ jobs:
plan:
- get: pipeline-source
trigger: true
passed: [operator-olm-build, docker-build]
passed: [docker-build]
- load_var: git-commit
file: pipeline-source/.git/short_ref
reveal: true
Expand Down Expand Up @@ -600,13 +600,19 @@ jobs:
INSTANA_ENDPOINT_HOST: ((instana-qa.endpoint_host))
INSTANA_ENDPOINT_PORT: 443
BUILD_BRANCH: ((branch))
INSTANA_API_KEY: ((qa-instana-api-token))
INSTANA_API_KEY: ((qa-instana-agent-key))
ARTIFACTORY_USERNAME: ((delivery-instana-io-internal-project-artifact-read-writer-creds.username))
ARTIFACTORY_PASSWORD: ((delivery-instana-io-internal-project-artifact-read-writer-creds.password))
inputs:
- name: pipeline-source
run:
path: pipeline-source/ci/scripts/end-to-end-test.sh
path: bash
args:
- -ceu
- |
cd pipeline-source
bash ./ci/scripts/cluster-authentication.sh
make e2e
on_success:
put: gh-status
inputs: [pipeline-source]
Expand Down Expand Up @@ -683,13 +689,19 @@ jobs:
INSTANA_API_URL: ((instana-qa.api_url))
INSTANA_API_TOKEN: ((instana-qa.api_token))
BUILD_BRANCH: ((branch))
INSTANA_API_KEY: ((qa-instana-api-token))
INSTANA_API_KEY: ((qa-instana-agent-key))
ARTIFACTORY_USERNAME: ((delivery-instana-io-internal-project-artifact-read-writer-creds.username))
ARTIFACTORY_PASSWORD: ((delivery-instana-io-internal-project-artifact-read-writer-creds.password))
inputs:
- name: pipeline-source
run:
path: pipeline-source/ci/scripts/end-to-end-test.sh
path: bash
args:
- -ceu
- |
cd pipeline-source
bash ./ci/scripts/cluster-authentication.sh
make e2e
on_success:
put: gh-status
inputs: [pipeline-source]
Expand Down Expand Up @@ -768,7 +780,7 @@ jobs:
# INSTANA_API_URL: ((instana-qa.api_url))
# INSTANA_API_TOKEN: ((instana-qa.api_token))
# BUILD_BRANCH: ((branch))
# INSTANA_API_KEY: ((qa-instana-api-token))
# INSTANA_API_KEY: ((qa-instana-agent-key))
# ARTIFACTORY_USERNAME: ((delivery-instana-io-internal-project-artifact-read-writer-creds.username))
# ARTIFACTORY_PASSWORD: ((delivery-instana-io-internal-project-artifact-read-writer-creds.password))
# inputs:
Expand Down
22 changes: 4 additions & 18 deletions ci/scripts/cluster-authentication.sh
Original file line number Diff line number Diff line change
Expand Up @@ -8,18 +8,6 @@
set -e
set -o pipefail

function cleanup_namespace() {
echo "Deleting the namespaces"
if kubectl get namespace/instana-agent ; then
kubectl delete namespace/instana-agent
kubectl wait --for=delete namespace/instana-agent --timeout=30s
echo "Deleted namespace instana-agent"
else
echo "Namespace instana-agent does not exist; skipping delete"
fi
echo "OK"
}

case "${CLUSTER_TYPE}" in
gke)
echo 'Testing on a GKE cluster'
Expand All @@ -42,19 +30,17 @@ case "${CLUSTER_TYPE}" in
export USE_GKE_GCLOUD_AUTH_PLUGIN=True
gcloud container clusters get-credentials "${GKE_CLUSTER_NAME}" --zone "${GKE_ZONE}" --project "${GKE_PROJECT}"

cleanup_namespace
;;
openshift)
echo "${KUBECONFIG_SOURCE}" > kubeconfig
chmod 0400 kubeconfig
KUBECONFIG="$(pwd)/kubeconfig"
export KUBECONFIG

cleanup_namespace

echo 'Download OpenShift CLI to modify rights for default ServiceAccount, so it has priviliged access'
curl -L --fail --show-error --silent https://mirror.openshift.com/pub/openshift-v4/clients/oc/latest/linux/oc.tar.gz -o /tmp/oc.tar.gz
tar -xzvf /tmp/oc.tar.gz --overwrite --directory /tmp
# go test does not require oc cli
# echo 'Download OpenShift CLI to modify rights for default ServiceAccount, so it has priviliged access'
# curl -L --fail --show-error --silent https://mirror.openshift.com/pub/openshift-v4/clients/oc/latest/linux/oc.tar.gz -o /tmp/oc.tar.gz
# tar -xzvf /tmp/oc.tar.gz --overwrite --directory /tmp

;;
*)
Expand Down
Loading

0 comments on commit c8e1573

Please sign in to comment.