Setting Up Store and Forward for EdgeX MQTT Export

[hdjoe]

I setup Store and Forward for the EdgeX MQTT Export Service; However, it is not storing data for forwarding whenever I disconnect it from the MQTT endpoint.

EdgeX Version: 2.3 (Levski)
MQTT Export Docker Image: edgexfoundry/app-service-configurable:2.3.0

• I went to the Consul and set the Enabled key value to true and left the RetryInterval and MaxRetryCount key values to 5m and 10 respectively. (key value path: /edgex/appservices/2.0/app-mqtt-export/Writable/StoreAndForward/)
  

• On the Consul, I also set the database host (Host) to point to the EdgeX Redis Container. (key value path: /edgex/appservices/2.0/app-mqtt-export/Database/Host)
  

• I then restarted the MQTT export container.

• I disconnected the MQTT export container from the MQTT endpoint for 30 minutes and observed in the logs that the export container was no longer able to reach the endpoint. However, I did not see any logs indicating that it was storing the data that was not transmitted.
  level=ERROR ts=2023-02-15T00:59:11.356795988Z app=app-mqtt-export source=messaging.go:199 msg="MessageBus Trigger: Failed to process message on pipeline(s): 1 error occurred:\n\t* in pipeline 'default-pipeline', unable to create MQTT Client: AuthModeCert selected however the key or cert PEM block was not found for secret=mqtt\n\n"

• I reconnected the MQTT export container to the MQTT endpoint.

• I observed that the MQTT export attempted to get the stored unsent data. But that query ended with no results.
  level=DEBUG ts=2023-02-15T01:29:07.595111921Z app=app-mqtt-export source=storeforward.go:142 msg="0 stored data items found for retrying"

I feel like I am missing a critical step that would inform the MQTT export to store the unsent data. I am not yet able to find any documentation on how this can be done. I am going through the code now, but If anyone has gotten past this, then I would appreciate the help.

Thanks

[lenny-goodell]

@hdjoe , there is one addition configuration of the MQTT Export, which is setting PersistOnError=true

app-mqtt-export/Writable/Pipeline/Functions/MQTTExport/Parameters/PersistOnError/

With this set you will see the following logged when the event can't be exported:

level=ERROR ts=2023-02-21T15:58:48.0340642Z app=app-mqtt-export source=triggermessageprocessor.go:186 msg="message error in pipeline default-pipeline (): in pipeline 'default-pipeline', could not connect to mqtt server for export, persisting Event for later retry. Error: network Error : dial tcp: lookup edgex-mqtt-broker on 127.0.0.11:53: no such host"

Add if you wait long enough before restarting the broker you will see:

level=ERROR ts=2023-02-21T15:59:45.6361884Z app=app-mqtt-export source=runtime.go:303 msg="Pipeline (default-pipeline) function #1 resulted in error: in pipeline 'default-pipeline', could not connect to mqtt server for export, persisting Event for later retry. Error: network Error : dial tcp: lookup edgex-mqtt-broker on 127.0.0.11:53: no such host (X-Correlation-ID=)"

Note I used a 30s retry interval to test with so I did have to wait 5 minutes to see the log messages.

Then when you restart the broker you will see:

level=DEBUG ts=2023-02-21T16:02:15.8171904Z app=app-mqtt-export source=mqttsecret.go:236 msg="Sent 391 bytes of data to MQTT Broker in pipeline 'default-pipeline'"
level=DEBUG ts=2023-02-21T16:02:15.8172063Z app=app-mqtt-export source=storeforward.go:147 msg=" 1 stored data items will be removed post retry"
level=DEBUG ts=2023-02-21T16:02:15.8172139Z app=app-mqtt-export source=storeforward.go:148 msg=" 0 stored data items will be update post retry"

And then after the next interval:

level=DEBUG ts=2023-02-21T16:03:45.8306015Z app=app-mqtt-export source=storeforward.go:142 msg="0 stored data items found for retrying"

[hdjoe]

Thx @lenny-intel
I am going to try this out.