Device Service does not update cache at first system boot

[SpAndrea117]

Hi everyone,

I'm trying to reproduce an EdgeX architecture where there are two device services. The first one handeles GPIO resources and the second one handles MODBUS readings. The architecture has been developed to trigger MODBUS readings only if a specific GPIO pin is up. To do so, i've added to device-modbus DS a device command triggered by the GPIO application service, that is responsible to send the device command to device-modbus. The whole flow works correctly, but at the first deployment of the application, both device services register to core metadata but their cache is not updated runtime, so to make the application works correctly i need to manually restart both. From the core metadata i've noticed the following logs:

level=ERROR ts=2022-10-06T10:18:54.323453816Z app=core-metadata source=notify.go:238 msg="fail to invoke device service callback for updating device service Modbus-Device, err: failed to send a http request -> Put \"http://device-modbus:59999/api/v2/callback/service\": dial tcp 10.43.64.87:59999: connect: no route to host" level=WARN ts=2022-10-06T10:18:54.323712223Z app=core-metadata source=notify.go:80 msg="Skipping device validation for device Modbus-Device (device service Modbus-Device unavailable)" level=ERROR ts=2022-10-06T10:18:54.537056091Z app=core-metadata source=notify.go:53 msg="fail to invoke device service callback for adding device Modbus-Device, err: failed to send a http request -> Post \"http://device-modbus:59999/api/v2/callback/device\": dial tcp 10.43.64.87:59999: connect: no route to host" level=WARN ts=2022-10-06T10:18:54.537120091Z app=core-metadata source=notify.go:80 msg="Skipping device validation for device Modbus-Device (device service Modbus-Device unavailable)" level=ERROR ts=2022-10-06T10:18:54.538079069Z app=core-metadata source=device.go:67 X-Correlation-ID=9ac78e17-2e4f-4105-b932-af18d77fe3f7 msg="device name Modbus-Device already exists" level=ERROR ts=2022-10-06T10:18:54.538242957Z app=core-metadata source=encoding.go:31 msg="Error encoding the data: http: Handler timeout" level=WARN ts=2022-10-06T10:19:16.972936249Z app=core-metadata source=notify.go:80 msg="Skipping device validation for device gpiod-Device (device service gpiod-Device unavailable)" level=ERROR ts=2022-10-06T10:19:17.188887542Z app=core-metadata source=notify.go:53 msg="fail to invoke device service callback for adding device gpiod-Device, err: failed to send a http request -> Post \"http://device-gpio:60000/api/v2/callback/device\": dial tcp 10.43.94.225:60000: connect: no route to host " level=ERROR ts=2022-10-06T10:24:52.94103392Z app=core-metadata source=notify.go:238 msg="fail to invoke device service callback for updating device service Modbus-Device, err: failed to send a http request -> Put \"http://device-modbus:59999/api/v2/callback/service\": dial tcp 10.43.64.87:59999: connect: no route to host"

The application deployment has been done with k3s and helm and i've tested all the possible network issues, so there should not be problem about this topic. I'm using the Kamakura version of EdgeX 2.2.0 and i've found the following issues about this problem:

edgexfoundry/device-sdk-go#376
edgexfoundry/device-virtual-go#56
edgexfoundry/device-sdk-go#376

Thanks

[cloudxxx8]

it's just the host setting issue. core-metadata can't communitcate with device service via REST callback
you need to set service host in the device service configuration with the proper hostname or ip address

[cloudxxx8]

although core-metadata can resolve the ip of device-gpiod, it can't reach the ip via http call according to the log you pasted

[SpAndrea117]

Hi, i've switched, fo device services, to the following modules in the go.mod file:

github.com/edgexfoundry/device-sdk-go/v2 v2.3.0-dev.37
github.com/edgexfoundry/go-mod-core-contracts/v2 v2.3.0-dev.18

In this case the logs change from no "route to host" to "connection refused", as the callback from metadata is called before the device service is ready to accept connections. I also have different behaviours for amd and arm architecture. In the first one, the cache seems to be updated runtime, meanwhile in ARM after the callback fail, from cache I always receive an empty list of devices associated to the service.

For the device service in the AMD case, the pod shows the following line:

level=ERROR ts=2022-10-10T08:01:18.152775218Z app=device-gpiod source=manageddevices.go:56 msg="failed to find Device device-gpiod in cache"
level=ERROR ts=2022-10-10T08:01:18.184582414Z app=device-gpiod source=init.go:76 msg="Failed to create the pre-defined devices: request failed, status code: 503, err: HTTP request timeout"

I've also tryed to remove the pod readiness probe, to make it available as it goes up, but the problem is still there.

Thanks

[SpAndrea117]

To add more context to the request I would like to add that i'm using the async channel to handle information between different services. For the autoevent use case i've read from a git issue that this problem does not persist. I would try to make an autoevent version, just to check if my issue is related to the async channel.

Thanks

[lenny-goodell]

@SpAndrea117 , this sounds like you are running in hybrid (docker/command-line) mode. Are you running the device services from command-line or in a container?

[SpAndrea117]

Hi,

I'm running all pods within the same kubernetes cluster that is a single node cluster on a Raspberry Pi 4. Each edgex service is executed by kubernetes in a single container pod.

[lenny-goodell]

Looks like device-modbus is resolving to 10.43.64.87. Is 59999 to correct port for Device Modbus? Default in the TOML file is 59901 unless you have changed it.

10-06T10:24:52.94103392Z app=core-metadata source=notify.go:238 msg="fail to invoke device service callback for updating device service Modbus-Device, err: failed to send a http request -> Put \"http://device-modbus:59999/api/v2/callback/service\": dial tcp 10.43.64.87:59999: connect: no route to host"

[SpAndrea117]

The device-modbus is a custom device service that i wrote from scratch. I need async readings triggered by a different service and i set Port 59999 for it.

Sorry for the name misunderstanding.