Issue With Loading a New Config Section from Consul

[tsconn23]

I am attempting to work on an integration between Project Alvarium and EdgeX Foundry. The first step is to integrate the Device SDK with the Alvarium SDK in order to capture metadata when new events are generated. I have performed this work in a local fork and further, I have consumed my version of the Device SDK in a fork of device-virtual. I am basing this work off the v3.0.0 tag in the upstream EdgeX repos.

The issue I am having is that I need to add a new Alvarium-specific section to the device service config. Since conceivably any device service would need to be Alvarium-enabled, I decided to modify the DeviceService section that is injected into Consul by edgex-core-common-config-bootstrapper. I have a Docker Compose file that I have created which unites all of the apps for EdgeX and Alvarium into one place for testing with security turned off.

When I start up the Docker Compose environment, the new section is available in Consul as expected.

However the Device Service container keeps crashing with an error that I know means the section is not being properly loaded into the service. Essentially the error means the value is blank.

level=INFO ts=2023-07-10T20:29:01.836268858Z app=device-virtual source=config.go:629 msg="Using Configuration provider (consul) from: http://edgex-core-consul:8500 with base path of edgex/v3/core-common-config-bootstrapper/all-services"
level=INFO ts=2023-07-10T20:29:01.847977189Z app=device-virtual source=config.go:353 msg="loading the common configuration for service type device-service"
level=INFO ts=2023-07-10T20:29:01.848665241Z app=device-virtual source=config.go:629 msg="Using Configuration provider (consul) from: http://edgex-core-consul:8500 with base path of edgex/v3/core-common-config-bootstrapper/device-services"
level=ERROR ts=2023-07-10T20:29:01.853554792Z app=device-virtual source=bootstrap.go:48 msg="failed to merge device-service config with common config: could not unmarshal JSON to *config.ConfigurationStruct: invalid KeyAlgorithm value provided "

To attempt to troubleshoot, I modified the Device Service to output the Alvarium section of its config when told to load from a local file. The output shows that the proper config is loaded.

export EDGEX_USE_REGISTRY=false
export EDGEX_SECURITY_SECRET_STORE=false

./device-virtual -cf configuration.yaml -r false -o true

level=INFO ts=2023-07-10T20:08:57.688201752Z app=device-virtual source=config.go:226 msg="Private configuration loaded from file with 0 overrides applied"
level=INFO ts=2023-07-10T20:08:57.693295011Z app=device-virtual source=variables.go:457 msg="Variables override of '-r/--registry' by environment variable: EDGEX_USE_REGISTRY=false"
level=INFO ts=2023-07-10T20:08:57.693615294Z app=device-virtual source=httpserver.go:133 msg="Web server starting (localhost:59900)"
level=INFO ts=2023-07-10T20:08:57.6936584Z app=device-virtual source=messaging.go:40 msg="MessageBus is disabled in configuration, skipping setup."
level=INFO ts=2023-07-10T20:08:57.693741617Z app=device-virtual source=metrics.go:64 msg="0 specified for metrics reporting interval. Setting to max duration to effectively disable reporting."
level=INFO ts=2023-07-10T20:08:57.69379205Z app=device-virtual source=manager.go:128 msg="Metrics Manager started with a report interval of 2562047h47m16.854775807s"
{"timestamp":"2023-07-10T20:08:57Z","hostname":"ubuntu-2","application":"./device-virtual","line-number":"init.go:60","log-level":"debug","message":"config loaded successfully"}
{"timestamp":"2023-07-10T20:08:57Z","hostname":"ubuntu-2","application":"./device-virtual","line-number":"init.go:61","log-level":"debug","message":"{\"annotators\":[\"tpm\",\"pki\"],\"hash\":{\"type\":\"sha256\"},\"signature\":{\"public\":{\"type\":\"ed25519\",\"path\":\"./res/keys/ed25519/public.key\"},\"private\":{\"type\":\"ed25519\",\"path\":\"./res/keys/ed25519/private.key\"}},\"stream\":{\"type\":\"mock\",\"config\":{\"provider\":{\"host\":\"localhost\",\"port\":8080,\"protocol\":\"http\"}}}}"}
level=INFO ts=2023-07-10T20:08:57.69468334Z app=device-virtual source=restrouter.go:57 msg="Registering v2 routes..."

The Device Service then fails for other reasons separate to this issue.

The logic involved in loading a config within the Device Service from an external provider is rather complex and I'm hoping someone has a quick insight into why this might be happening -- perhaps something simple I've overlooked. If not, my next step is to stand up Consul natively and manually bootstrap the common config. After that I will then configure my IDE to allow me to step through and attempt to isolate the problem.

Thanks for any bright ideas one might have.

[lenny-goodell]

@tsconn23 , did you add the Alvarium structure to the Go SDK's configuration struct here: https://github.com/edgexfoundry/device-sdk-go/blob/main/internal/config/config.go#L15-L31 so it has fields to marshal into?

[tsconn23]

@lenny-intel Yes, got that covered in my local fork.

https://github.com/tsconn23/device-sdk-go/blob/e9caad90d02720bf9c2cfbbfc5eff95ee2e37d80/internal/config/config.go#L18

[lenny-goodell]

Also, the -cf flag is for when not using Consul so you can specifiy a file that contains the Common Config. The -o flag is only needed if you previously ran the service in Docker and need to override the services private config in Consul.

Use ./device-virtual -cp -d
See:

• https://docs.edgexfoundry.org/3.0/microservices/configuration/CommonCommandLineOptions/#config-provider
• https://docs.edgexfoundry.org/3.0/microservices/configuration/CommonCommandLineOptions/#developer-mode
• https://docs.edgexfoundry.org/3.0/microservices/configuration/CommonCommandLineOptions/#common-config

Also see Common Config description here: https://docs.edgexfoundry.org/3.0/microservices/configuration/CommonConfiguration/

[lenny-goodell]

@tsconn23 , the Annotators slice here is the cause the error loading the configuration. This is because array types are not supported when using Consul. This has been a limitation since Intel joined the project due to the https://github.com/mitchellh/consulstructure module used when loading the configuration from Consul. Push to Consul works fine, but loading from doesn't.

To confirm, try removing the Annotators from your common config.

[tsconn23]

@lenny-intel Actually, I don't think that's the problem. I've been stepping through the config init logic in the device-virtual this morning and I think I've root caused it.

What I found was that everything in the Alvarium structure WAS populated except for the Alvarium.Signature section. Why would that be? In the YAML config (and thus in Consul) the keys corresponding to these properties are public and private whereas the property names are PublicKey and PrivateKey. I have used YAML struct tags to map the keys to the properties and indeed, when I load the config file from disk, the properties ARE populated. Not so in Consul.

I notice that these two properties appear to be the only ones in the entire config who alias is substantively different from their proper name. I interpret this to mean that when the consulstructure.Decoder() is mapping keys from Consul to the ConfigurationStruct that it's ignoring any JSON/YAML mapping and is instead using a literal mapping. That could produce undesirable behavior and is something to be aware of.

The fix is for me to change the configuration.yaml seeded by core-common-config-bootstrapper to use publicKey and privateKey for the respective YAML entries. When I do this, the struct is populated as expected.

Note that in the screenshot I've expanded the Annotators array to show that it's populated correctly.

I'm going to run another test with this change through the Docker Compose scenario but I'm expecting that I'll be able to close this issue shortly.

[lenny-goodell]

Yep, when don't JSON annotate our config structs, so the the config YAML must match the struct field name (upper or lower case)

I will be surprised if the slice works... But that issue may have only been with the Writable section.

[tsconn23]

@lenny-intel It sounds like you previously ran into an issue with how arrays defined in the config are hydrated at runtime. Is this the error you encountered?

level=ERROR ts=2023-07-11T18:18:34.89449474Z app=device-virtual source=bootstrap.go:48 msg="failed to merge device-service config with common config: could not unmarshal JSON to *config.ConfigurationStruct: json: cannot unmarshal string into Go struct field ConfigurationStruct.Alvarium of type int"

Look at the beginning of the debug output below and notice how the annotators property value is wrapped in brackets...followed by quotes...followed by brackets again. Is this the issue you're thinking of?

I don't see how this is solvable off the top of my head, at least not if I need to use Consul. The steps as I understand this flow are as follows:

1.) I construct a config in YAML which is used to populate Consul
2.) Consul is queried to hydrate an internal "device" ConfigurationStruct via KV literals
3.) That "device" config is then merged into a "source" ConfigurationStruct via JSON Marshal/Unmarshal

Given three different methodologies for mapping, I'm not surprised something is lost in translation. I'm unsure what to do at this point. My thinking is the only possible way to get this to work without Alvarium SDK changes specifically befitting EdgeX is to run the services natively using local configs without any config provider at all. But that sounds like a lot of work.

[lenny-goodell]

Yea, something like that, but not sure where the type int is coming from.
You can verify that Annotators is the issue by deleting Annotators key from Consul, start the service and verify error goes away.

This has nothing to do with the mappings. This issue is that https://github.com/mitchellh/consulstructure never supported slices.

The work around (hack) is to not use the Alvarium SDK's struct for the config. Define a struct in the Device SDK that handles Annotators using a map, i.e. map[string]int. Then translate the local config into the Alvarium SDK struct when it is needed. i.e convert the map keys to the []string

Use the Annotators strings as the map keys in the YAML

annotators:
   tpm: 0
   pki: 0

Note that we have an task for this dev cycle (Napa) to look at a redesign of the Consul implementation to fix an issue with the Writable section complicated by how https://github.com/mitchellh/consulstructure works. This will result in no longer using https://github.com/mitchellh/consulstructure for Writable and then why using it for pull... Implementation would be post Napa.

[tsconn23]

I'm going to go ahead and close this discussion. I'm not exactly sure what I'm going to do about it yet, but I think we've root caused the issue.