Implement M3 client

[rjb1000]

Feature description

Using OpenAPI YAML interface definition files, implement an M3 client for the 5GMSd AF that communicates with one or more instances of the 5GMSd AS developed in #30.

• Provision and manage Server Certificate resources available to the test client as local X.509 PEM files.
  • Create, Retrieve, Update, Destroy.
• Provision and manage Content Hosting Configuration resources available to the test client as local JSON files.
  • Create, Retrieve, Update, Destroy.
  • Purge the associated content cache.

Note that there is nothing to stimulate the destruction of Content Hosting Configurations and server certificates in the Application Function as part of the scope of the M3 Link project. (Destruction only comes into play when the M1 provisioning is implemented.) Nevertheless, the destruction operations should still be implemented as part of the scope of this feature. These operations can be tested by placing a fake event on the Open5GS event loop.

Relevant specifications and corresponding sections

• TS 26.512.
• Pre-standardisation M3 Open API YAML interface definition files - see Define M3 APIs rt-common-shared#12.

Additional context

In the first instance, the 5GMSd AF only knows about a single Content Hosting Configuration, loaded from a local JSON file, but passive provision should be made for it to maintain provisioning state for one Content Hosting Configuration per Provisioning Session, indexed by provisioningSessionId.

The JSON configuration file for the 5GMSd AF needs to be enhanced to list the FQDN or IP address of the 5GMSd AS that it is managing. Passive provision should be made in this configuration to list multiple 5GMSd AS instances.

Implementation design

Due to the event driven nature of Open5GS NFs the design is split into multiple events and actions which operate on application context of the 5GMS Application Function. To make this simple at first, we will start with a mapping of the provisioning session content hosting configuration to one application server, i.e. the application server deals with all distribution configurations in the content hosting configuration. The application context will therefore hold an array of application server state structures which will each hold:

• A pointer to the application server configuration this state is related to.
• A list of provisioning sessions assigned to the application server.
• A, possibly uninitialised, list of certificates currently known to the application server.
• A list of certificates that need uploading (adding or updating) on the application server.
• A list of certificates that need deleting from the application server.
• A, possibly uninitialised, list of content hosting configurations currently known to the application server.
• A list of content hosting configurations to upload (add or update) on the application server.
• A list of content hosting configurations to delete from the application server.

When a ContentHostingConfiguration arrives

Ultimately this will happen on the M1 interface, but for now this is triggered by loading the configured JSON file.

1. Parse the ContentHostingConfiguration and check for syntax correctness (done by openapi-generator created bindings)
2. Verify the ContentHostingConfiguration and raise an error if the configuration is incorrect
   i. Check that the application function has been configured with the certificates referenced by any distributionConfigurations.certificateId fields.
3. Choose an application server to implement this configuration (for now just take the first on the list, we will add some load balancing later).
4. For each distributionConfigurations.certificateId, add the provisioningSessionId + ':' + distributionConfigurations.certificateId to the application server state list of certificates to upload.
5. Add the provisioningSessionId for the ContentHostingConfiguration to the application server state list of content hosting configurations to upload.
6. Call a function to perform the next M3 operation for the chosen application server.

Function to perform the next M3 operation for a given application server state

This will perform one step in synchronising the application server with the desired state (lists of certificates and content hosting configurations to upload or delete).

1. If the current known certificates list in the application server state is uninitialised
   1. Send a GET /certificates request to the application server.
   2. Return
2. If there are entries in the upload certificates list in the application server state
   1. If the first entry in the upload certificates list also appears in the currently known certificates list
      1. Send the certificate referred to by the first entry from the upload certificates list using a PUT /certificates/<certificateId> to the application server.
   2. Otherwise
      1. Send the certificate referred to by the first entry from the upload certificates list using a POST /certificates/<certificateId> to the application server.
   3. Return
3. If the current known content hosting configurations list in the application server state is uninitialised
   1. Send a GET /content-hosting-configurations request to the application server.
   2. Return
4. If there are entries in the upload content hosting configurations list in the application server state
   1. If the first entry in the upload content hosting configurations list also appears in the currently known content hosting configurations list
      1. Send the content hosting configuration referred to by the provisioning session Id from the first entry from the upload content hosting configurations list using a PUT /content-hosting-configurations/<provisioningSessionId> to the application server.
   2. Otherwise
      1. Send the content hosting configuration referred to by the provisioning session Id from first entry from the upload content hosting configurations list using a POST /content-hosting-configurations/<provisioningSessionId> to the application server.
   3. Return
5. If there are entries in the delete content hosting configurations list in the application server state
   1. Using the id from the first entry in the delete content hosting configurations list, send DELETE /content-hosting-configurations/<id> to the application server.
   2. Return
6. If there are entries in the delete certificates list in the application server state
   1. Using the id from the first entry in the delete certificates list, send DELETE /certificates/<id> to the application server.
   2. Return
7. Return

When a client response is received

The following actions assume 2XX responses, if an error response is received from an application server it should be dealt with appropriately.

1. If the request was for GET /certificates
   1. Parse the body of the response as a JSON array and return an error if bad syntax.
   2. Initialise or empty the known certificates list in the application server state associated with the request
   3. Populate the list with the array entries
   4. Perform the next M3 operation for the application server state
2. If the request was for POST /certificates/<id>
   1. Remove the entry for id from the upload certificates list in the application server state associated with the request
   2. Add the entry for id to the current known certificates list in the application server state associated with the request
   3. Perform the next M3 operation for the application server state
3. If the request was for PUT /certificates/<id>
   1. Remove the entry for id from the upload certificates list in the application server state associated with the request
   2. Perform the next M3 operation for the application server state
4. If the request was for DELETE /certificates/<id>
   1. Remove the entry for id from the delete certificates list in the application server state associated with the request
   2. Remove the entry for id from the current known certificates list in the application server state associated with the request
   3. Perform the next M3 operation for the application server state
5. If the request was for GET /content-hosting-configurations
   1. Parse the body of the response as a JSON array and return an error if bad syntax.
   2. Initialise or empty the known content hosting configurations list in the application server state associated with the request
   3. Populate the list with the array entries
   4. Perform the next M3 operation for the application server state
6. If the request was for POST /content-hosting-configurations/<id>
   1. Remove the entry for id from the upload content hosting configurations list in the application server state associated with the request
   2. Add the entry for id to the current known content hosting configurations list in the application server state associated with the request
   3. Perform the next M3 operation for the application server state
7. If the request was for PUT /content-hosting-configurations/<id>
   1. Remove the entry for id from the upload content hosting configurations list in the application server state associated with the request
   2. Perform the next M3 operation for the application server state
8. If the request was for DELETE /content-hosting-configurations/<id>
   1. Remove the entry for id from the delete content hosting configurations list in the application server state associated with the request
   2. Remove the entry for id from the current known content hosting configurations list in the application server state associated with the request
   3. Perform the next M3 operation for the application server state

This design means that any wanted updates to an application server's state can be set in the upload and delete lists in the specific application server state, and the appropriate operations will be performed in sequence in order to achieve that state. At the end of which the upload and delete lists should be empty and no more actions will be performed.

[devbbc]

Implementation of the Client is almost done. Can send requests but haven't tested the responses with the Application server.

[devbbc]

The Application Function now has the implementation of the upload and current lists of both the Content Hosting Configuration and the certificates. This enables the Application Function to send state update requests to the application server via the M3 interface.

This is an example output of M3 requests from the AF captured with netcat:

PUT /3gpp-m3/v1/certificates/d54a1fcc-d411-4e32-807b-2c60dbaeaf5f:certificate-1.pem HTTP/1.1
Host: localhost:8080
Accept: */*
Content-Type: application/x-pem-file
Content-Length: 2823

-----BEGIN PRIVATE KEY-----
...

PUT /3gpp-m3/v1/content_hosting_configurations/d54a1fcc-d411-4e32-807b-2c60dbaeaf5f HTTP/1.1
Host: localhost:8080
Accept: */*
Content-Type: application/json
Content-Length: 711

{
	"distributionConfigurations":	[{
			"canonicalDomainName":	"localhost",
			"domainNameAlias":	"",
			"pathRewriteRules":	[{
					"mappedPath":	"/datasets/DASHDataset2014/BigBuckBunny/4sec/",
					"requestPattern":	"^/m4d/provisioning-session-[^/]*/"
				}]
		}, {
			"canonicalDomainName":	"localhost",
			"certificateId":	"testcert1",
			"domainNameAlias":	"",
			"pathRewriteRules":	[{
					"mappedPath":	"/datasets/DASHDataset2014/BigBuckBunny/4sec/",
					"requestPattern":	"^/m4d/provisioning-session-[^/]*/"
				}]
		}],
	"ingestConfiguration":	{
		"entryPoint":	"https://ftp.itec.aau.at/",
		"protocol":	"urn:3gpp:5gms:content-protocol:http-pull-ingest",
		"pull":	true
	},
	"name":	"Big Buck Bunny"
}

The implementation of the delete lists is still pending.

[devbbc]

Next steps:

• Address some minor bugs that have been identified in the request messages.
• Integration testing with the Application Server to check that the M3 responses are processed correctly.

[devbbc]

The above example output shows:
PUT /3gpp-m3/v1/certificates/d54a1fcc-d411-4e32-807b-2c60dbaeaf5f:certificate-1.pem HTTP/1.1

In this case, the 5GMS AF uses this ContentHostingConfiguration, where the certificate id testcert1. The corresponding certificate name is certificate-1.pem.

The request header should be of the form: /3gpp-m3/v1/certificates/provisioningSessionId:certificateId.
So, in our example output, the header should have testcert1 instead of certificate-1.pem.

Modified the code, so that it now has the certificate Id. The output of the certificate request with the modifications captured with netcat:

POST /3gpp-m3/v1/certificates/d54a1fcc-d411-4e32-807b-2c60dbaeaf5f:testcert1 HTTP/1.1
Host: localhost:8080
Accept: */*
Content-Type: application/x-pem-file
Content-Length: 2823

-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDHLwUQUeFEK0w/
8KnpL8NNlYJNLCtjoqvJXAeWiTC52xzUU2PEGULZQaCAP1lwNKWR7Ocvo88kYF0G
7Hfs5xG9Rah70rgrZ9iXWFcSESxKJpkxjjDQQmtI6emL9iH8NFoS62hAaG9kOHmX
U2c9u/cURuMxZWwVYt+bm4BitmoF1bjVFleFnU88VAlHgG7cAYOnYI20ouUUDsT2
....

[devbbc]

Implemented the delete operation in the M3 client.

Started testing the AF's M3 client with the corresponding server in the Application Server. The client request for certificates and the servers responds with status 200.

Further tests and analysis need to be done to establish the full operations.

[devbbc]

Have been testing the Application Function against the Application Server.

• The Application Function can GET certificates and content-hosting-configurations from the Application server. It initialises the upload & current lists and adds certificates, if appropriate. It then POSTs the certificate to the Application Server.
• The Application server seems to create / add the certificate successfully as it tries to send response with status 201. But it crashes when sending the response (see Application Server crash on certificate upload rt-5gms-application-server#46). As a result, the Application Function receives Internal Server error.
• Same error happens when trying to update the certificate through the PUT method.

[rjb1000]

Discussed with @devbbc how to test the destruction operations in the absence of an external stimulus. Ticket #8 discusses some options. In this case, option 1 (inotify) sounds most promising. The deletion of the Content Hosting Configuration file could be the stimulus to test the destruction operation. The Application Function will need to cross-reference the Content Hosting Configuration's resource identifier with the file in order to destroy the correct resource when the corresponding file is deleted.

It's not a problem if this is a Linux-only OS feature, since it's only used for testing and can be removed or commented out later.

[devbbc]

Implemented a method to detect the deletion of the Content Hosting Configuration file using inotify APIs. The Application Function now has context for inotify as well as a data structure to store the Content Hosting Configuration's resource identifier and the corresponding Content Hosting Configuration file. With the help of this data structure, the Application Function can now cross-reference the Content Hosting Configuration's resource identifier with the corresponding configuration file.

Yet to test the delete operation with this stimulus.

[devbbc]

The code to PUSH the certificates and Content Hosting Configuration has been successfully tested with the Application Server.

The test shows that the certificate and Content Hosting Configuration appear in the AS and being configured up in the Nginx.

Last thing to do is the delete operation.

[devbbc]

Successfully tested delete operation with the Application Server for deletion of both the Content Hosting Configuration and Certificate. On response 204 from the Application Server, the Application Function destroys them from their relevant lists.

[davidjwbbc]

After the update to the M3 YAML files in 5G-MAG/rt-common-shared#14, the generated APIs will have changed slightly. Most of these changes do not affect the AF implementation as the client API is manually implemented and is not affected by the changes, or the changes are in an unimplemented API (purge).

The one that will affect the AF is that the M3 Certificates APIs no longer use {provisioningSessionId}:{certificateId} and now use a single {afUniqueCertificateId}

• This doesn't affect the certificates operation of the Application Function as it can implement the afUniqueCertificateId as "{provisioningSessionId}:{certificateId}" anyway as its choice of unique ID.
• This will affect the pushing of ContentHostingConfigurations to the AS via M3 as any certificateIds in the distribution configurations now need to be sent to the AS with the provisioning session id prefixed. So when a CHC is to be sent the AF needs to:
  • OpenAPI_content_hosting_configuration_copy() the original CHC to a new modified CHC.
  • Where an entry in the distribution_configurations list of the modified CHC has a certificate_id set:
    • Create a new string which is "{provisioning_session_id}:{certificate_id}"
    • Free the old certificate_id in the list entry and replace it with the new string.
  • Send the modified CHC with a PUT or POST (as was done before with the original CHC).
  • Delete the modified CHC with OpenAPI_content_hosting_configuration_free() once the request has been sent.

This will abstract away how the unique certificate Id is generated by the AF so that the AS only needs to know it will be unique and it will match the distribution configurations certificate Ids sent to the AS in the ContentHostingConfiguration.

[devbbc]

The Application Function considers "{provisioningSessionId}:{certificateId}" as its afUniqueCertificateId.

Modified the code so that any certificateIds in the distribution configurations are now prefixed with the Provisioning Session Id, when a contentHostingConfiguration is to be sent to the AF.

Tested the working of the M3 interface by running this Application Function against the M3 server of the Application Server. They worked as expected.