Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

initdata: migrate key release test cases to initdata #2006

Merged
merged 5 commits into from
Sep 2, 2024

Conversation

huoqifeng
Copy link

@huoqifeng huoqifeng commented Aug 15, 2024

Fixes: #1985

  • Migrate key release test cases to use initdata
  • Remove AA_KBC_PARAMS and pkg/aa, pkg/cdh etc.
  • Update the documents for AA_KBC_PARAMS
  • Added INITDATA in cm peer-pods-cm for default initdata for every Pod, Annotation in Pod will overwrite the settings in INITDATA.

because aa, cdh certificates might need kbs upgrade, so certificates and others support will be in separate PRs.

@huoqifeng huoqifeng marked this pull request as draft August 15, 2024 09:43
@huoqifeng
Copy link
Author

Tested for libvirt provider against sample TEE:

# export TEST_PROVISION_FILE="/root/libvirt.properties"
export CLOUD_PROVIDER=libvirt
export DEPLOY_KBS=true
export TEST_KBS=true
export TEST_INSTALL_CAA=yes
export TEST_TEARDOWN=no
make test-e2e
go test -v -tags=libvirt -timeout 60m -count=1 -run '' ./test/e2e
time="2024-08-15T09:35:26Z" level=info msg="Do setup"
time="2024-08-15T09:35:26Z" level=info msg="Container runtime: containerd"
time="2024-08-15T09:35:26Z" level=info msg="Deploying kbs"
time="2024-08-15T09:35:26Z" level=info msg="creating key.bin"
time="2024-08-15T09:35:26Z" level=info msg="Creating kbs install overlay"
time="2024-08-15T09:35:27Z" level=info msg="Customize the overlay yaml file"
time="2024-08-15T09:35:27Z" level=info msg="Updating kbs image with \"ghcr.io/confidential-containers/staged-images/kbs\""
time="2024-08-15T09:35:27Z" level=info msg="Updating kbs image tag with \"e890fc90c384207668fa3a4d6a2f2a2d652797ee\""
time="2024-08-15T09:35:27Z" level=info msg="Creating kbs install overlay"
time="2024-08-15T09:35:27Z" level=info msg="Install Kbs"
Wait for the kbs deployment be available
time="2024-08-15T09:35:32Z" level=info msg="kbsEndpoint: http://192.168.122.233:32670"
time="2024-08-15T09:35:32Z" level=info msg="Install Cloud API Adaptor"
time="2024-08-15T09:35:32Z" level=info msg="Deploy the Cloud API Adaptor"
time="2024-08-15T09:35:32Z" level=info msg="Install the controller manager"
Wait for the cc-operator-controller-manager deployment be available
time="2024-08-15T09:35:49Z" level=info msg="Customize the overlay yaml file"
time="2024-08-15T09:35:50Z" level=info msg="Install the cloud-api-adaptor"
Wait for the cc-operator-daemon-install DaemonSet be available
Wait for the cloud-api-adaptor-daemonset DaemonSet be available
Wait for the pod cloud-api-adaptor-daemonset-5zgwx be ready
Wait for the kata-remote runtimeclass be created
time="2024-08-15T09:36:40Z" level=info msg="Installing peerpod-ctrl"
time="2024-08-15T09:36:41Z" level=info msg="Wait for the peerpod-ctrl deployment to be available"
time="2024-08-15T09:36:56Z" level=info msg="Creating namespace 'coco-pp-e2e-test-316f6bdb'..."
time="2024-08-15T09:36:56Z" level=info msg="Wait for namespace 'coco-pp-e2e-test-316f6bdb' be ready..."
time="2024-08-15T09:37:01Z" level=info msg="Wait for default serviceaccount in namespace 'coco-pp-e2e-test-316f6bdb'..."
time="2024-08-15T09:37:01Z" level=info msg="default serviceAccount exists, namespace 'coco-pp-e2e-test-316f6bdb' is ready for use"
=== RUN   TestLibvirtKbsKeyRelease
time="2024-08-15T09:37:01Z" level=info msg="set key resource: ../../kbs/config/kubernetes/overlays/x86_64/key.bin"
time="2024-08-15T09:37:01Z" level=info msg="EnableKbsCustomizedPolicy: ../../kbs/sample_policies/allow_all.rego"
time="2024-08-15T09:37:01Z" level=info msg="EnableKbsCustomizedPolicy: ../../kbs/sample_policies/deny_all.rego"
=== PAUSE TestLibvirtKbsKeyRelease
=== CONT  TestLibvirtKbsKeyRelease
    common_suite.go:604: Do test kbs key release failure case
=== RUN   TestLibvirtKbsKeyRelease/DoTestKbsKeyReleaseForFailure_test
    assessment_runner.go:265: Waiting for containers in pod: kbs-failure are ready
=== RUN   TestLibvirtKbsKeyRelease/DoTestKbsKeyReleaseForFailure_test/Kbs_key_release_is_failed
=== NAME  TestLibvirtKbsKeyRelease
    common_suite.go:623: PASS as failed to access key.bin: 
=== NAME  TestLibvirtKbsKeyRelease/DoTestKbsKeyReleaseForFailure_test/Kbs_key_release_is_failed
    assessment_runner.go:421: Output when execute test commands: 
=== NAME  TestLibvirtKbsKeyRelease/DoTestKbsKeyReleaseForFailure_test
    assessment_runner.go:607: Deleting pod kbs-failure...
    assessment_runner.go:614: Pod kbs-failure has been successfully deleted within 60s
=== NAME  TestLibvirtKbsKeyRelease
    libvirt_test.go:33: KBS normal cases
time="2024-08-15T09:38:51Z" level=info msg="EnableKbsCustomizedPolicy: ../../kbs/sample_policies/allow_all.rego"
    common_suite.go:580: Do test kbs key release
=== RUN   TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test
    assessment_runner.go:265: Waiting for containers in pod: kbs-key-release are ready
=== RUN   TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test/Kbs_key_release_is_successful
=== NAME  TestLibvirtKbsKeyRelease
    common_suite.go:588: Success to get key.bin: This is my cluster name: 
=== NAME  TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test/Kbs_key_release_is_successful
    assessment_runner.go:421: Output when execute test commands: This is my cluster name: 
=== NAME  TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test
    assessment_runner.go:607: Deleting pod kbs-key-release...
    assessment_runner.go:614: Pod kbs-key-release has been successfully deleted within 60s
--- PASS: TestLibvirtKbsKeyRelease (215.24s)
    --- PASS: TestLibvirtKbsKeyRelease/DoTestKbsKeyReleaseForFailure_test (109.48s)
        --- PASS: TestLibvirtKbsKeyRelease/DoTestKbsKeyReleaseForFailure_test/Kbs_key_release_is_failed (9.44s)
    --- PASS: TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test (105.54s)
        --- PASS: TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test/Kbs_key_release_is_successful (5.49s)
PASS
ok  	github.com/confidential-containers/cloud-api-adaptor/src/cloud-api-adaptor/test/e2e	310.005s

@huoqifeng
Copy link
Author

I'll retest after the refactoring, I guess we can start reviewing also.

@huoqifeng huoqifeng marked this pull request as ready for review August 15, 2024 14:18
@huoqifeng
Copy link
Author

Test and passed:

  • Build the caa image and set it in libvirt overlay
# cat install/overlays/libvirt/kustomization.yaml 
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- ../../yamls

images:
- name: cloud-api-adaptor
  newName: quay.io/huoqif/cloud-api-adaptor
  newTag: dev-3feaa8899997453e34c203f2d84a71ea87ccc8b0
  • Build the podvm and upload to vol
  • Create a properties file
# cat /root/libvirt.properties 
libvirt_uri="qemu+ssh://[email protected]/system?no_verify=1"
libvirt_ssh_key_file="id_rsa"
CLUSTER_NAME = "peer-pods"
KBS_IMAGE = "ghcr.io/confidential-containers/staged-images/kbs"
KBS_IMAGE_TAG = "e890fc90c384207668fa3a4d6a2f2a2d652797ee"
  • Run the key release test for libvirt against sample tee
export TEST_PROVISION_FILE="/root/libvirt.properties"
export CLOUD_PROVIDER=libvirt
export DEPLOY_KBS=true
export TEST_KBS=true
export TEST_INSTALL_CAA=yes
export TEST_TEARDOWN=no
make test-e2e
go test -v -tags=libvirt -timeout 60m -count=1 -run '' ./test/e2e
time="2024-08-16T02:46:51Z" level=info msg="Do setup"
time="2024-08-16T02:46:51Z" level=info msg="Container runtime: containerd"
time="2024-08-16T02:46:51Z" level=info msg="Deploying kbs"
time="2024-08-16T02:46:51Z" level=info msg="creating key.bin"
time="2024-08-16T02:46:51Z" level=info msg="Creating kbs install overlay"
time="2024-08-16T02:46:51Z" level=info msg="Customize the overlay yaml file"
time="2024-08-16T02:46:51Z" level=info msg="Updating kbs image with \"ghcr.io/confidential-containers/staged-images/kbs\""
time="2024-08-16T02:46:52Z" level=info msg="Updating kbs image tag with \"e890fc90c384207668fa3a4d6a2f2a2d652797ee\""
time="2024-08-16T02:46:52Z" level=info msg="Creating kbs install overlay"
time="2024-08-16T02:46:52Z" level=info msg="Install Kbs"
Wait for the kbs deployment be available
time="2024-08-16T02:46:57Z" level=info msg="kbsEndpoint: http://192.168.122.233:30489"
time="2024-08-16T02:46:57Z" level=info msg="Install Cloud API Adaptor"
time="2024-08-16T02:46:57Z" level=info msg="Deploy the Cloud API Adaptor"
time="2024-08-16T02:46:57Z" level=info msg="Install the controller manager"
Wait for the cc-operator-controller-manager deployment be available
time="2024-08-16T02:47:14Z" level=info msg="Customize the overlay yaml file"
time="2024-08-16T02:47:15Z" level=info msg="Install the cloud-api-adaptor"
Wait for the pod cloud-api-adaptor-daemonset-gsl7p be ready
Wait for the kata-remote runtimeclass be created
time="2024-08-16T02:48:05Z" level=info msg="Installing peerpod-ctrl"
time="2024-08-16T02:48:06Z" level=info msg="Wait for the peerpod-ctrl deployment to be available"
time="2024-08-16T02:48:21Z" level=info msg="Creating namespace 'coco-pp-e2e-test-8114a144'..."
time="2024-08-16T02:48:21Z" level=info msg="Wait for namespace 'coco-pp-e2e-test-8114a144' be ready..."
time="2024-08-16T02:48:26Z" level=info msg="Wait for default serviceaccount in namespace 'coco-pp-e2e-test-8114a144'..."
time="2024-08-16T02:48:26Z" level=info msg="default serviceAccount exists, namespace 'coco-pp-e2e-test-8114a144' is ready for use"
=== RUN   TestLibvirtKbsKeyRelease
time="2024-08-16T02:48:26Z" level=info msg="set key resource: ../../kbs/config/kubernetes/overlays/x86_64/key.bin"
time="2024-08-16T02:48:26Z" level=info msg="EnableKbsCustomizedPolicy: ../../kbs/sample_policies/allow_all.rego"
time="2024-08-16T02:48:26Z" level=info msg="EnableKbsCustomizedPolicy: ../../kbs/sample_policies/deny_all.rego"
=== PAUSE TestLibvirtKbsKeyRelease
=== CONT  TestLibvirtKbsKeyRelease
    common_suite.go:604: Do test kbs key release failure case
=== RUN   TestLibvirtKbsKeyRelease/DoTestKbsKeyReleaseForFailure_test
    assessment_runner.go:265: Waiting for containers in pod: kbs-failure are ready
=== RUN   TestLibvirtKbsKeyRelease/DoTestKbsKeyReleaseForFailure_test/Kbs_key_release_is_failed
=== NAME  TestLibvirtKbsKeyRelease
    common_suite.go:623: PASS as failed to access key.bin: 
=== NAME  TestLibvirtKbsKeyRelease/DoTestKbsKeyReleaseForFailure_test/Kbs_key_release_is_failed
    assessment_runner.go:421: Output when execute test commands: 
=== NAME  TestLibvirtKbsKeyRelease/DoTestKbsKeyReleaseForFailure_test
    assessment_runner.go:607: Deleting pod kbs-failure...
    assessment_runner.go:614: Pod kbs-failure has been successfully deleted within 60s
=== NAME  TestLibvirtKbsKeyRelease
    libvirt_test.go:35: KBS normal cases
time="2024-08-16T02:50:16Z" level=info msg="EnableKbsCustomizedPolicy: ../../kbs/sample_policies/allow_all.rego"
    common_suite.go:580: Do test kbs key release
=== RUN   TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test
    assessment_runner.go:265: Waiting for containers in pod: kbs-key-release are ready
=== RUN   TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test/Kbs_key_release_is_successful
=== NAME  TestLibvirtKbsKeyRelease
    common_suite.go:588: Success to get key.bin: This is my cluster name: 
=== NAME  TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test/Kbs_key_release_is_successful
    assessment_runner.go:421: Output when execute test commands: This is my cluster name: 
=== NAME  TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test
    assessment_runner.go:607: Deleting pod kbs-key-release...
    assessment_runner.go:614: Pod kbs-key-release has been successfully deleted within 60s
--- PASS: TestLibvirtKbsKeyRelease (214.98s)
    --- PASS: TestLibvirtKbsKeyRelease/DoTestKbsKeyReleaseForFailure_test (109.32s)
        --- PASS: TestLibvirtKbsKeyRelease/DoTestKbsKeyReleaseForFailure_test/Kbs_key_release_is_failed (9.28s)
    --- PASS: TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test (105.46s)
        --- PASS: TestLibvirtKbsKeyRelease/KbsKeyReleasePod_test/Kbs_key_release_is_successful (5.41s)
PASS
ok  	github.com/confidential-containers/cloud-api-adaptor/src/cloud-api-adaptor/test/e2e	309.741s

@huoqifeng huoqifeng added the test_e2e_libvirt Run Libvirt e2e tests label Aug 16, 2024
@huoqifeng
Copy link
Author

@mkulke may you help check whether the changes are OK for azure?

Copy link
Member

@stevenhorsman stevenhorsman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally it looks good to me. A couple of points that would make the review/future commit checks easier:

  • Adding commit body explaining the why/how of the changes
  • Squashing some of the commits together e.g. the current 4th commit and quite a lot of the 3rd commit, could be fixed up into the first commit for more clarity and grouping of the code.

Thanks for the great work!

src/cloud-api-adaptor/test/e2e/common.go Outdated Show resolved Hide resolved
@bpradipt
Copy link
Member

Fixes: #1985

  • Migrate key release test cases to use initdata
  • Remove AA_KBC_PARAMS and pkg/aa, pkg/cdh etc.
  • Update the documents for AA_KBC_PARAMS

because aa, cdh certificates might need kbs upgrade, so certificates and others support will be in separate PRs.

Just re-thinking about it from a ux standpoint. Is it really needed to remove the support for AA_KBC_PARAMS ?
Can't initdata (via annotation) override the default AA_KBC_PARAMS that's set as part of the infra?

@huoqifeng
Copy link
Author

Just re-thinking about it from a ux standpoint. Is it really needed to remove the support for AA_KBC_PARAMS ? Can't initdata (via annotation) override the default AA_KBC_PARAMS that's set as part of the infra?

@bpradipt any specific reason to keep AA_KBC_PARAMS? Keep AA_KBC_PARAMS will increase the maintain efforts for userdata and adpator package. Also, it might introduce different behavior. For example: It does not support certificates in aa and cdh toml viaAA_KBC_PARAMS and hard to add (We might not want add also).

@mkulke
Copy link
Contributor

mkulke commented Aug 17, 2024

Just re-thinking about it from a ux standpoint. Is it really needed to remove the support for AA_KBC_PARAMS ? Can't initdata (via annotation) override the default AA_KBC_PARAMS that's set as part of the infra?

That would be another configuration that we'd have to support and test. beyond a URI AA_KBC_PARAMs doesn't support certs or any current or future option for AA and CDH. I suspect AA_KBC_PARAMS will also be retired for kata-qemu (at least for cc_kbc). If we want to have a per-installation configuration for CAA to improve the UX, I suggest to (optionally) provide a default initdata.toml o via configmap, similar to the auth.json and use that one if no initdata is set as annotation on a pod.

@huoqifeng
Copy link
Author

right, more config are adding in cdh.toml like image pull and secure mount besides the certificates. Looks just keeping AA_KBC_PARAMs does not meet the requirements. A global initdata which can be overwrote by Pod's annotation (either in configmap or in secret) might be a good approach.

@bpradipt
Copy link
Member

@huoqifeng @mkulke thanks for clarifying. Providing default initData via configMap is a good option.
@huoqifeng is this something you want to handle as part of the overall AA_KBC_PARAMS removal work ?

@huoqifeng
Copy link
Author

@huoqifeng @mkulke thanks for clarifying. Providing default initData via configMap is a good option. @huoqifeng is this something you want to handle as part of the overall AA_KBC_PARAMS removal work ?

It was not in my original plan, I added it as an item in #1985, hopefully, I can have time to do that.

Qi Feng Huo added 2 commits August 20, 2024 14:46
 - migrate key release test cases to initdata
 - remove AA_KBC_PARAMS and aaKBCParams
 - use allow-all rego policy to make key release test run correctly

Fixes: confidential-containers#1985

Signed-off-by: Qi Feng Huo <[email protected]>
- add global-initdata in configmap and parameters
Fixes: confidential-containers#1985

Signed-off-by: Qi Feng Huo <[email protected]>
@huoqifeng
Copy link
Author

@huoqifeng @mkulke thanks for clarifying. Providing default initData via configMap is a good option. @huoqifeng is this something you want to handle as part of the overall AA_KBC_PARAMS removal work ?

@bpradipt the GLOBAL_INITDATA added in configmap peer-pods-cm to replace AA_KBC_PARAMS, may you please have a look? CC @mkulke

Copy link
Contributor

@mkulke mkulke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only a few comments, I'm not able to test the code currently. It would be good to also have a look at the current kata-qemu PR for init-data, since that PR is using the agent's SetInitData endpoint and this would probably overlap with what CAA is doing atm

src/cloud-api-adaptor/docs/initdata.md Show resolved Hide resolved
src/cloud-api-adaptor/pkg/adaptor/cloud/cloud.go Outdated Show resolved Hide resolved
src/cloud-api-adaptor/pkg/adaptor/cloud/cloud.go Outdated Show resolved Hide resolved
src/cloud-api-adaptor/pkg/adaptor/server.go Outdated Show resolved Hide resolved
- rename GLOBAL_INITDATA to INITDATA
- rename CdhFilePath to CDHConfigPath
- rename AaFilePath to AAConfigPath

Fixes: confidential-containers#1985

Signed-off-by: Qi Feng Huo <[email protected]>
@huoqifeng
Copy link
Author

huoqifeng commented Aug 21, 2024

only a few comments, I'm not able to test the code currently. It would be good to also have a look at the current kata-qemu PR for init-data, since that PR is using the agent's SetInitData endpoint and this would probably overlap with what CAA is doing atm

I'll keep monitoring the PR in kata-qemu. kata-containers/kata-containers#10163

- Validate the initdata passed in both from configmap and annotation
Fixes: confidential-containers#1985

Signed-off-by: Qi Feng Huo <[email protected]>
@huoqifeng
Copy link
Author

if invalid initdata used, the error looks like:

2024/08/21 09:05:44 [adaptor/cloud] initdata in Pod annotation: 
2024/08/21 09:05:44 [adaptor/cloud] initdata in pod annotation is empty, use global initdata: aWJ2YWxpZCA9IGRkZA==
2024/08/21 09:05:44 [adaptor/cloud] Error unmarshalling initdata: toml: incomplete number

Copy link
Member

@bpradipt bpradipt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm
Thanks @huoqifeng

@huoqifeng huoqifeng merged commit 289c0a1 into confidential-containers:main Sep 2, 2024
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
test_e2e_libvirt Run Libvirt e2e tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Replace AA_KBC_PARAMS after enable initdata
4 participants