Create standard for mandatory and supported IaaS services

[josephineSei]

closes SovereignCloudStack/issues#528

[markus-hentsch]

Some suggestions for minor rephrasing of some things inline.

[markus-hentsch]

Also one thing I only addressed in the review in a shallow manner so far:

I believe we discussed in the community calls previously that we need to make a clearer distinction between OpenStack APIs and the services and be careful about the wording in standards.

The OpenStack services are essentially reference implementations of the APIs. Wherever possible, the SCS should mandate the APIs and their functionalities, not the (backend) services. CSPs should be free to use whatever implementation they like as long as it provides the same API and behavior¹. Especially when mandating components, this can be a make or break deal for CSPs.

Footnotes

1. not that I expect anybody to implement something like a full Nova alternative overnight but from a formal standpoint, the freedom should be there ↩

[anjastrunk]

Two cosmetic issues and one question, which requires clarification.

[artificial-intelligence]

the service lists themselves mostly LGTM, but I think some other parts could need some clarification, see the comments.

Thanks for working on this.

[josephineSei]

I think we should discuss the supported services list in the IaaS call, as this topic is very important and should not be merged without having overall agreement also for the supported service list.

[josephineSei]

We discussed this PR today in the IaaS Call. Especially what "supported service" means.

• What should users expect from these supported services?
  • that standards refering to overall IaaS also include them (== all services can be enabled to fulfill the standards?)
  • that these services have been tested for integration?
  • that these services are part of the reference implementation? (imho that might be too much)
    • "supported" in the standardization sense can not make statements on the reference implementation
    • meaning of "supported" in standardization may be limited to ensuring that our standards don't conflict with these
      • we don't break them with our standards!
      • renaming "supported" to "recommended"? No: "recommended" has a stronger meaning ...
      • we originally wanted to separate between
        1. (mandatory) you need these APIs to have a scs-compliant cloud
        2. (supported) you may use these APIs, they can be integrated with scs (e.g. work with the scs role standard)
        3. (unsupported) you may use these APIs on your own risk
  • hints and recommendations for the implementation may be added to the implementation notes
• Generic discussion: We do standardize concrete APIs (the ones that come from OpenStack or K8s or CAPI) to make the
  standards useful -- alternative implementations for services are possible, but a completely different set of technologies
  will unlikely implement the SCS APIs.
• We need to use the correct terminology "OpenStack Compute API" as opposed to "Nova"
  • Complexity: These APIs are huge, with lots of optional and deprecated pieces
    • Need to work through these, tedious
    • The OpenStack Interop Guideline tests ("OpenStack powered XXX") are a good starting point for this
      • This is a fairly small set, we need to add to this to produce something that is really useful ("this is the motivation for SCS standardization")

In conclusion the PR should focus mor on APIs than OpenStack services. Which has the downside, that we would also need to clarify for each API endpoint, whether this is mandatory, recommended or optional / not needed.

This is a direction which is worth going for, but this will add a load of complexity and might need specific documents for each service API.

[artificial-intelligence]

I think I found mostly some typos and some missing replacements of /services/APIs/.

[josephineSei]

When I'm correct the only thing missing here for this standard is:
Is the Heat API still in the supported list or not?
@garloff and @fkr you wanted to discuss this, right? Do you have any news about it?

[anjastrunk]

An other question came to my mind: How does a user know, which API version a certain SCS cloud supports? There is a document on certificates listing all effective standards for a certain scope and version. Standard on mandatory services will be part of a scopes/version, soon. How does the user/reader figure out, which API of which OpenStack service belongs to a given certificate scope and version.

[josephineSei]

As discussed in the IaaS meeting, we want to standardize, that the s3 should be mandatory and swift only supported.
I looking into the s3 protocol and the hints for tests from Kurt to adjust the current testing behavior.

The one open question to me (which needs to be tested) is how do we get an endpoint for s3 when we do not have swift? or do we require that swift should be always there - even just as a stump to provide that endpoint in the service catalog?

[josephineSei]

I was able to test my script and Kurts script against a deployment from Cloud&Heat, which provides a Swift endpoint and also offers S3 behind this endpoint.

$ python3 mandatory-iaas-services.py --os-cloud openstack
The following endpoints are missing: ['block-storage', 'load-balancer']

But it also came up, that there was another description for "block-storage" that was just volume. I need to adjust the test, to allow both options - or do we want to only allow one specific endpoint name:

1. do we require that all endpoints have to show up in catalog list output?
2. do we want to maybe only allow one name per service (only block-storage OR volume for the block storage service)?

I would like to answer the first question with yes, because in this way we will always have access to the object storage endpoint, regardless, what service is used.

The second question is more difficult: we may need to adjust the tests to allow more than one name per service - each time a new name comes up. On the other hand: that will have a huge impact on CSPs, so I don't think, we should strictly only allow one name.

[josephineSei]

I am installing a minIO server on the same machine my devstack runs on to create a setup, which has an OpenStack and an s3 on the same level but next to it and no Swift enabled. I am looking into minIO to create a user and credentials, which i can use for the test.

[josephineSei]

I installed minio in a test environment and started it temporarily:

root@devstack:/opt/stack/devstack# ./minio server /data
MinIO Object Storage Server
Copyright: 2015-2024 MinIO, Inc.
License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html
Version: RELEASE.2024-09-22T00-33-43Z (go1.22.7 linux/amd64)

API: http://192.168.23.161:9000  http://192.168.11.28:9000  http://10.0.2.1:9000  http://192.168.122.1:9000  http://127.0.0.1:9000 
   RootUser: minioadmin 
   RootPass: minioadmin 

WebUI: http://192.168.23.161:46367 http://192.168.11.28:46367 http://10.0.2.1:46367 http://192.168.122.1:46367 http://127.0.0.1:46367          
   RootUser: minioadmin 
   RootPass: minioadmin 

CLI: https://min.io/docs/minio/linux/reference/minio-mc.html#quickstart
   $ mc alias set 'myminio' 'http://192.168.23.161:9000' 'minioadmin' 'minioadmin'

Docs: https://docs.min.io
WARN: Detected default credentials 'minioadmin:minioadmin', we recommend that you change these values with 'MINIO_ROOT_USER' and 'MINIO_ROOT_PASSWORD' environment variables

In another console tab I installed the minio client, logged in as admin, created a new user and gave that users rights to read and write:

stack@devstack:~/devstack$ curl https://dl.min.io/client/mc/release/linux-amd64/mc --create-dirs -o $HOME/minio-binaries/mc
stack@devstack:~/devstack$ mc alias set 'myminio' 'http://192.168.23.161:9000' 'minioadmin' 'minioadmin'
Command 'mc' not found, but can be installed with:
apt install mc
Please ask your administrator.
stack@devstack:~/devstack$ chmod +x $HOME/minio-binaries/mc
stack@devstack:~/devstack$ export PATH=$PATH:$HOME/minio-binaries/
stack@devstack:~/devstack$ mc alias set 'myminio' 'http://192.168.23.161:9000' 'minioadmin' 'minioadmin'
Added `myminio` successfully.
stack@devstack:~/devstack$ mc admin user add myminio newuser newusersecret
Added user `newuser` successfully.
stack@devstack:~/devstack$ mc admin user list myminio
enabled    newuser                                   
stack@devstack:~/devstack$ mc admin policy list myminio
consoleAdmin
diagnostics
readonly
readwrite
writeonly
stack@devstack:~/devstack$ mc admin policy attach myminio readwrite --user newuser
Attached Policies: [readwrite]
To User: newuser

After this I was able to test the script and made adjustments to it until I reached the point, where it succeeded:

stack@devstack:~/devstack$ python3 mandatory-iaas-services3.py --os-cloud mycloud2 --s3-endpoint "http://192.168.23.161:9000" --s3-access newuser --s3-access-secret newusersecret
FAIL: The following endpoints are missing: ['load-balancer']
SUCCESS: S3 exists

When not giving the parameters it will fail:

stack@devstack:~/devstack$ python3 mandatory-iaas-services3.py --os-cloud mycloud2
FAIL: The following endpoints are missing: ['load-balancer', 'object-store']
FAIL: No object store endpoint found. No testing for the s3 service possible. Details: public endpoint for object-store service not found

[josephineSei]

@gtema could you review this standard again? Do you have time to do this? If not can you just give a small notice?

[josephineSei]

@fkr or @garloff do you want to take another look at this?