app-service-configurable metrics-influxdb does not recieve metrics as corresponding redis key does not exist

[b4zs]

I am trying to use app-service-configurable with the metrics-influxdb profile to push data to InfluxDB.
I have the basic docker setup with device-virtual enabled, and I see its data stream on the GUI (DataCenter/Reading).
As far as I understand this profile subscribes to edgex/telemetry/# and forwards data via HTTP, but this topic does not exist in the database.
Listing the redis keys returns a long list, each starting with patterns like "cd|rd", "md|dp", "ss|iv", etc, but not one named "edgex". Isn't the messagebus readable via redis-cli or what am I missing?
Thanks.

[jiekechoo]

@b4zs What is the EdgeX version? and any details about your environment？such as: app configuration, InfluxDB configuration, ...

[b4zs]

I am using version 3.1, Napa, with the generated docker compose file, that uses the edgexfoundry/service-name:3.1.0 images.
In case of InfluxDB, I am using the latest version.

  influxdb:
    image: influxdb:latest
    container_name: influxdb
    networks:
      - default
      - edgex-network
    security_opt:
      - no-new-privileges:true
    restart: unless-stopped
    ports:
      - "8086:8086"
    volumes:
      - ./var/influxdb2/db:/var/lib/influxdb2

I do not think that this is related to InfluxDB, as it is the exporter service that can not subscribe to the corresponding topic:

  app-influxdb-export:
    image: edgexfoundry/app-service-configurable:3.1.0
    container_name: app-influxdb-export
    hostname: app-influxdb-export
    depends_on:
      - consul
      - core-data
    environment:
      EDGEX_PROFILE: metrics-influxdb # @see https://github.com/edgexfoundry/app-service-configurable/blob/v3.1.0/res/metrics-influxdb/configuration.yaml
      CLIENTS_CORE_COMMAND_HOST: edgex-core-command
      CLIENTS_CORE_DATA_HOST: edgex-core-data
      CLIENTS_CORE_METADATA_HOST: edgex-core-metadata
      CLIENTS_SUPPORT_NOTIFICATIONS_HOST: edgex-support-notifications
      CLIENTS_SUPPORT_SCHEDULER_HOST: edgex-support-scheduler
      MESSAGEQUEUE_HOST: edgex-redis
      REGISTRY_HOST: edgex-core-consul
      SERVICE_HOST: app-influxdb-export
      DATABASE_HOST: edgex-redis
      EDGEX_SECURITY_SECRET_STORE: "false"
      WRITABLE_PIPELINE_FUNCTIONS_HTTPEXPORT_PARAMETERS_URL: "http://app-debug-http:8080/api/v2/write?org=sample-org&bucket=${INFLUX_BUCKET}&precision=ns"
      WRITABLE_INSECURESECRETS_INFLUXDB_SECRETDATA_TOKEN: "Token ${INFLUX_TOKEN}"
    networks:
      - default
      - edgex-network
    ports:
      - 127.0.0.1:59708:59707/tcp
    read_only: true
    security_opt:
      - no-new-privileges:true

(no extra configuration file mounted)

I have also tried to send the data to a custom HTTP service (mendhak/http-https-echo), this is why the HTTPEXPORT_PARAMETERS_URL points to app-debug-http.
Is there any way to monitor the internal message bus to check whether the telemetry/# topic has content?

Thank you.

[jiekechoo]

I am trying to use app-service-configurable with the metrics-influxdb profile to push data to InfluxDB.

Sorry, I don't get your point.

But,

• Here is the metrics-influxdb doc

• Here is the Influxdb Export

Which one do you want?

[lenny-goodell]

Listing the redis keys returns a long list, each starting with patterns like "cd|rd", "md|dp", "ss|iv", etc, but not one named "edgex". Isn't the messagebus readable via redis-cli or what am I missing?

@b4zs , Edgex uses Redis Pub/Sub not Streams. The topics will not show up in keys.

Is there any way to monitor the internal message bus to check whether the telemetry/# topic has content?

See link above. You should be able to subscribe via the CLI using PSUBSCRIBE to edgex.telemetry.*

Note the change in syntaxe for level separator and wildcard when using the Redis CLI

[lenny-goodell]

I am using version 3.1, Napa, with the generated docker compose file, that uses the edgexfoundry/service-name:3.1.0 images.

You app-metrics-influxdb YAML above is a mix of 3.1 and 2,x yaml. Here is what it should look like for Napa:

  app-metrics-influxdb:
    container_name: edgex-app-metrics-influxdb
    depends_on:
      consul:
        condition: service_started
      core-data:
        condition: service_started
    environment:
      EDGEX_PROFILE: metrics-influxdb
      EDGEX_SECURITY_SECRET_STORE: "false"
      SERVICE_HOST: edgex-app-metrics-influxdb
    hostname: edgex-app-metrics-influxdb
    image: edgexfoundry/app-service-configurable:3.1.0
    networks:
      edgex-network: null
    ports:
    - mode: ingress
      host_ip: 127.0.0.1
      target: 59707
      published: "59707"
      protocol: tcp
    read_only: true
    restart: always
    security_opt:
    - no-new-privileges:true
    user: 2002:2001
    volumes:
    - type: bind
      source: /etc/localtime
      target: /etc/localtime
      read_only: true
      bind:
        create_host_path: true

I have attached the compose file from running make gen no-secty asc-metrics ds-virtual
docker-compose.yml.txt

[lenny-goodell]

What do the logs for the app-metrics-influxdb service show?
If no errors, then add WRITABE_LOGLEVEL: DEBUG and verify the service if receiving data. If not DEBUG messages, then no data is being received to be exported..

[jiekechoo]

If set DEBUG mode, no error logs.

level=INFO ts=2024-02-06T00:57:12.889332742Z app=app-<profile> source=service.go:523 msg="Starting app-metrics-influxdb 3.1.0 "
level=INFO ts=2024-02-06T00:57:12.889473399Z app=app-metrics-influxdb source=config.go:718 msg="Using Configuration provider (consul) from: http://edgex-core-consul:8500 with base path of edgex/v3/core-common-config-bootstrapper/all-services"
level=WARN ts=2024-02-06T00:57:17.32228713Z app=app-metrics-influxdb source=config.go:1056 msg="did not get boolean from config provider for edgex/v3/core-common-config-bootstrapper/IsCommonConfigReady: strconv.ParseBool: parsing \"\": invalid syntax"
level=WARN ts=2024-02-06T00:57:17.322337367Z app=app-metrics-influxdb source=config.go:1064 msg="waiting for Common Configuration to be available from config provider"
level=WARN ts=2024-02-06T00:57:18.323923337Z app=app-metrics-influxdb source=config.go:1056 msg="did not get boolean from config provider for edgex/v3/core-common-config-bootstrapper/IsCommonConfigReady: strconv.ParseBool: parsing \"\": invalid syntax"
level=WARN ts=2024-02-06T00:57:18.32396875Z app=app-metrics-influxdb source=config.go:1064 msg="waiting for Common Configuration to be available from config provider"
level=INFO ts=2024-02-06T00:57:19.32938851Z app=app-metrics-influxdb source=config.go:422 msg="loading the common configuration for service type app-service"
level=INFO ts=2024-02-06T00:57:19.329697685Z app=app-metrics-influxdb source=config.go:718 msg="Using Configuration provider (consul) from: http://edgex-core-consul:8500 with base path of edgex/v3/core-common-config-bootstrapper/app-services"
level=INFO ts=2024-02-06T00:57:19.338284363Z app=app-metrics-influxdb source=config.go:173 msg="Common configuration loaded from the Configuration Provider. No overrides applied"
level=INFO ts=2024-02-06T00:57:19.338338713Z app=app-metrics-influxdb source=config.go:718 msg="Using Configuration provider (consul) from: http://edgex-core-consul:8500 with base path of edgex/v3/app-metrics-influxdb"
level=INFO ts=2024-02-06T00:57:19.339487325Z app=app-metrics-influxdb source=variables.go:462 msg="Variables override of '-p/-profile' by environment variable: EDGEX_PROFILE=metrics-influxdb"
level=INFO ts=2024-02-06T00:57:19.339573395Z app=app-metrics-influxdb source=config.go:731 msg="Loading configuration file from res/metrics-influxdb/configuration.yaml"
level=INFO ts=2024-02-06T00:57:19.340221077Z app=app-metrics-influxdb source=variables.go:462 msg="Variables override of 'Service/Host' by environment variable: SERVICE_HOST=edgex-app-metrics-influxdb"
level=INFO ts=2024-02-06T00:57:19.340264467Z app=app-metrics-influxdb source=variables.go:462 msg="Variables override of 'Writable/Pipeline/Functions/HTTPExport/Parameters/Url' by environment variable: WRITABLE_PIPELINE_FUNCTIONS_HTTPEXPORT_PARAMETERS_URL=http://192.168.123.12:8086/api/v2/write?org:edgex&bucket:edgex&precision:ns"
level=INFO ts=2024-02-06T00:57:19.340292129Z app=app-metrics-influxdb source=variables.go:462 msg="Variables override of 'Writable/InsecureSecrets/influxdb/SecretData/token' by environment variable: WRITABLE_INSECURESECRETS_INFLUXDB_SECRETDATA_TOKEN=Token mrTd06ETr08IN4VLDczosVC1D1Sbbi9VQUuoTh_NW71rarg9HFpI2m_oIBpGwBFgOFQB47VUZB_BTPXCGl988w=="
level=INFO ts=2024-02-06T00:57:19.340312664Z app=app-metrics-influxdb source=variables.go:462 msg="Variables override of 'Writable/LogLevel' by environment variable: WRITABLE_LOGLEVEL=DEBUG"
level=INFO ts=2024-02-06T00:57:19.34033118Z app=app-metrics-influxdb source=config.go:251 msg="Private configuration loaded from file with 4 overrides applied"
level=INFO ts=2024-02-06T00:57:19.35951749Z app=app-metrics-influxdb source=config.go:262 msg="Private configuration has been pushed to into Configuration Provider with overrides applied"
level=INFO ts=2024-02-06T00:57:19.359548415Z app=app-metrics-influxdb source=config.go:269 msg="listening for private config changes"
level=INFO ts=2024-02-06T00:57:19.359560952Z app=app-metrics-influxdb source=config.go:271 msg="listening for all services common config changes"
level=INFO ts=2024-02-06T00:57:19.35957745Z app=app-metrics-influxdb source=config.go:274 msg="listening for application service common config changes"
level=INFO ts=2024-02-06T00:57:19.35959926Z app=app-metrics-influxdb source=registry.go:61 msg="Using Registry access token of length 0"
level=INFO ts=2024-02-06T00:57:19.359619702Z app=app-metrics-influxdb source=registry.go:89 msg="Using Registry (consul) from http://edgex-core-consul:8500"
level=INFO ts=2024-02-06T00:57:19.364692534Z app=app-metrics-influxdb source=messaging.go:66 msg="Setting options for secure MessageBus with AuthMode='usernamepassword' and SecretName='redisdb"
level=INFO ts=2024-02-06T00:57:19.364748612Z app=app-metrics-influxdb source=messaging.go:104 msg="Connected to redis Message Bus @ redis://edgex-redis:6379 with AuthMode='usernamepassword'"
level=WARN ts=2024-02-06T00:57:19.364975664Z app=app-metrics-influxdb source=clients.go:180 msg="unable to Get service endpoint for 'core-metadata': no matching service endpoint found. retrying..."
level=INFO ts=2024-02-06T00:57:20.365959105Z app=app-metrics-influxdb source=clients.go:190 msg="Using registry for URL for 'core-metadata': http://edgex-core-metadata:59881"
level=DEBUG ts=2024-02-06T00:57:20.367272918Z app=app-metrics-influxdb source=version.go:134 msg="Confirmed Core Services version (3.1.0) is compatible with SDK's version (v3.1.0)"
level=INFO ts=2024-02-06T00:57:20.367316753Z app=app-metrics-influxdb source=metrics.go:64 msg="0 specified for metrics reporting interval. Setting to max duration to effectively disable reporting."
level=INFO ts=2024-02-06T00:57:20.367349185Z app=app-metrics-influxdb source=manager.go:128 msg="Metrics Manager started with a report interval of 2562047h47m16.854775807s"
level=INFO ts=2024-02-06T00:57:20.36737024Z app=app-metrics-influxdb source=bootstrap.go:251 msg="SecuritySecretsRequested metric registered and will be reported (if enabled)"
level=INFO ts=2024-02-06T00:57:20.367384494Z app=app-metrics-influxdb source=bootstrap.go:251 msg="SecuritySecretsStored metric registered and will be reported (if enabled)"
level=INFO ts=2024-02-06T00:57:20.367459572Z app=app-metrics-influxdb source=server.go:57 msg="Registering standard routes..."
level=INFO ts=2024-02-06T00:57:20.367476862Z app=app-metrics-influxdb source=server.go:75 msg="Registering CORS middleware..."
level=INFO ts=2024-02-06T00:57:20.367488471Z app=app-metrics-influxdb source=service.go:587 msg="Service started in: 7.478222111s"
level=INFO ts=2024-02-06T00:57:20.367514159Z app=app-metrics-influxdb source=main.go:38 msg="Loading Configurable Pipeline..."
level=INFO ts=2024-02-06T00:57:20.367501519Z app=app-metrics-influxdb source=configupdates.go:48 msg="Waiting for App Service configuration updates..."
level=DEBUG ts=2024-02-06T00:57:20.3675349Z app=app-metrics-influxdb source=service.go:308 msg="Default Function Pipeline Execution Order: [ToLineProtocol, Batch, HTTPExport]"
level=DEBUG ts=2024-02-06T00:57:20.367680191Z app=app-metrics-influxdb source=service.go:401 msg="ToLineProtocol function added to 'default-pipeline' configurable pipeline with parameters: [tags='']"
level=DEBUG ts=2024-02-06T00:57:20.367731093Z app=app-metrics-influxdb source=service.go:401 msg="Batch function added to 'default-pipeline' configurable pipeline with parameters: [batchthreshold='100', mergeonsend='true', mode='bytimecount', timeinterval='60s']"
level=DEBUG ts=2024-02-06T00:57:20.36778425Z app=app-metrics-influxdb source=service.go:401 msg="HTTPExport function added to 'default-pipeline' configurable pipeline with parameters: [headername='authorization', method='post', mimetype='raw', secretname='influxdb', secretvaluekey='token', url='http://192.168.123.12:8086/api/v2/write?org:edgex&bucket:edgex&precision:ns']"
level=DEBUG ts=2024-02-06T00:57:20.367805227Z app=app-metrics-influxdb source=main.go:46 msg="Found 1 configured pipeline(s)"
level=INFO ts=2024-02-06T00:57:20.367871267Z app=app-metrics-influxdb source=runtime.go:197 msg="PipelineMessagesProcessed-default-pipeline metric has been registered and will be reported (if enabled)"
level=INFO ts=2024-02-06T00:57:20.367907222Z app=app-metrics-influxdb source=runtime.go:197 msg="PipelineMessageProcessingTime-default-pipeline metric has been registered and will be reported (if enabled)"
level=INFO ts=2024-02-06T00:57:20.36792549Z app=app-metrics-influxdb source=runtime.go:197 msg="PipelineProcessingErrors-default-pipeline metric has been registered and will be reported (if enabled)"
level=INFO ts=2024-02-06T00:57:20.367948095Z app=app-metrics-influxdb source=runtime.go:122 msg="Transforms set for `default-pipeline` pipeline"
level=DEBUG ts=2024-02-06T00:57:20.367968578Z app=app-metrics-influxdb source=service.go:419 msg="Default pipeline added with 3 transform(s)"
level=INFO ts=2024-02-06T00:57:20.368001174Z app=app-metrics-influxdb source=triggermessageprocessor.go:90 msg="MessagesReceived metric has been registered and will be reported"
level=INFO ts=2024-02-06T00:57:20.36802096Z app=app-metrics-influxdb source=triggermessageprocessor.go:96 msg="InvalidMessagesReceived metric has been registered and will be reported (if enabled)"
level=INFO ts=2024-02-06T00:57:20.368035395Z app=app-metrics-influxdb source=triggerfactory.go:51 msg="EdgeX MessageBus trigger selected"
level=INFO ts=2024-02-06T00:57:20.368056696Z app=app-metrics-influxdb source=messaging.go:64 msg="Initializing EdgeX Message Bus Trigger for 'redis'"
level=INFO ts=2024-02-06T00:57:20.368078958Z app=app-metrics-influxdb source=messaging.go:84 msg="Subscribing to topic: edgex/telemetry/#"
level=INFO ts=2024-02-06T00:57:20.368091628Z app=app-metrics-influxdb source=messaging.go:94 msg="Publish topic not set for Trigger. Response data, if set, will not be published"
level=INFO ts=2024-02-06T00:57:20.368120293Z app=app-metrics-influxdb source=server.go:168 msg="Starting HTTP Web Server on address edgex-app-metrics-influxdb:59707"
level=INFO ts=2024-02-06T00:57:20.368155766Z app=app-metrics-influxdb source=messaging.go:102 msg="Waiting for messages from the MessageBus on the 'edgex/telemetry/#' topic"
level=INFO ts=2024-02-06T00:57:20.36892646Z app=app-metrics-influxdb source=service.go:241 msg="StoreAndForward disabled. Not running retry loop."
level=INFO ts=2024-02-06T00:57:20.368950097Z app=app-metrics-influxdb source=service.go:244 msg="app-metrics-influxdb has Started"

I set mqtt-export, it works.

Nothing in the InfluxDB:

[b4zs]

I have updated my configurations with the provided yaml, checked redis with the mentioned command (thanks for pointing out that i did not use pub/sub) but there are still no channels in redis:

127.0.0.1:6379> PUBSUB CHANNELS
(empty array)

Isn't it necessary to enable these communication channels manually? I still see a lot of keys being updated continuously (MONITOR redis command) as the virtual devices provide their data, but nothing in pub/sub.

However, I confirm that mqtt-export works perfectly, which is sufficient for my use case, as I can forward data to InfluxDB with Telegraf, but this is something I wanted to avoid.

[lenny-goodell]

You don't have any metrics data being published. Have you enabled any of the metrics? They are all disabled by defaults.
https://docs.edgexfoundry.org/3.1/microservices/general/#service-metrics

Which service are you wanting the service metrics from?

There are a few common security metrics
https://docs.edgexfoundry.org/3.1/microservices/configuration/CommonConfiguration/#common-configuration-properties
The rest are App Service, Device Service or service specific. See Writable.Telemetry tab at following links
https://docs.edgexfoundry.org/3.1/microservices/application/GeneralAppServiceConfig/
https://docs.edgexfoundry.org/3.1/microservices/device/Ch-DeviceServices/#configuration-properties
https://docs.edgexfoundry.org/3.1/microservices/core/data/Configuration/

[lenny-goodell]

However, I confirm that mqtt-export works perfectly, which is sufficient for my use case, as I can forward data to InfluxDB with Telegraf, but this is something I wanted to avoid.

MQTT Export works because it is receiving Event data which it can export, not Metrics data. You obviously have a device service running that is generating the Event data.

[b4zs]

I managed to get metrics logging to work using the provided documentation. I completely missed the corresponding part for this configuration. Thank you for your patient support. I realized that I did not provide sufficient information on my use case and configuration.

Details:
I am using the provided docker compose service configuration for "app-metrics-influxdb", influxdb, device-virtual to provide test data, and the "edgexfoundry/device-mqtt" service to collect device events from MQTT. I wanted to log the metrics of the "device-mqtt" service to Influx.

Now that I added Writable.Telemetry.Metrics.ReadingsSent to the configuration of the "device-mqtt" service, metrics reporting is showing up in the logs of the service:

edgex-device-mqtt  | level=DEBUG ts=2024-02-06T17:17:53.339400638Z app=device-mqtt source=reporter.go:195 msg="Publish 1 metrics to the 'edgex/telemetry/device-mqtt' base topic"
edgex-device-mqtt  | level=DEBUG ts=2024-02-06T17:17:53.339520388Z app=device-mqtt source=manager.go:123 msg="Reported metrics..."

Also, the app-metrics-influxdb service is able to send these metrics to InfluxDB:

edgex-app-metrics-influxdb  | level=INFO ts=2024-02-06T17:01:35.264990212Z app=app-metrics-influxdb source=helpers.go:47 msg="HttpExportErrors-http://influxdb:8086/api/v2/write?org=sample-org&bucket=edgex-data&precision=ns metric has been registered and will be reported (if enabled)"
edgex-app-metrics-influxdb  | level=INFO ts=2024-02-06T17:01:35.265379379Z app=app-metrics-influxdb source=helpers.go:47 msg="HttpExportSize-http://influxdb:8086/api/v2/write?org=sample-org&bucket=edgex-data&precision=ns metric has been registered and will be reported (if enabled)"

The data shows up in InfluxDB, thus my requirements are met.
PSUBSCRIBE edgex.telemetry.* also works in redis-cli.

Thank you.