Kuiper with EdgeX security

[neerajasjawali]

Hi all,

We are working on an EdgeX (jakarta) application where we use eKuiper rules engine to actuate a device. When a GPIO IR sensor detects a moment of person, then the kuiper send a command to a usb device to start capturing.
We want to implement the same with EdgeX security services.

Any suggestions on how to proceed integrating kuiper with EdgeX security services?

Thanks in advance

[bnevis-i]

Any suggestions on how to proceed integrating kuiper with EdgeX security services?

What do you mean by this? Kuiper --is-- integrated with EdgeX running in secure mode.

Can you explain the delta between the current behavior running EdgeX in secure mode and the expected behavior?

[ngjaying]

The log seems fine. Can you diagnose if this topic edgex/events/#/Random-UnsignedInteger-Device/Uint16 do have data?

[neerajasjawali]

Yes. When subscribed to the topic, I am receiving the data.

[neerajasjawali]

Does the secret tokens generated for the app-rules-engine applies for the kuiper?

[ngjaying]

If no secret token, I don't think the subscription can be successful. From the log, eKuiper just cannot receive any data from that topic. @lenny-intel Any ideas how to diagnose?

[neerajasjawali]

Hi, @ngjaying @bnevis-i @lenny-intel

Modifications done in order to connect kuiper with security services. So, kindly suggest me how do I proceed with the testing.

Contents of /etc/kuiper/sources/edgex.yaml

#Global Edgex configurations           
default:                                             
  protocol: redis                                                    
  #protocol: tcp                                                             
  server: edgex-redis                                                
  #port: 5563                                                                
  port: 6379                           
  connectionSelector: edgex.redisMsgBus              
  topic: rules-events                                                
  type: redis                                                                
  # Could be 'event' or 'request'.                                   
  # If the message is from app service, the message type is an event;        
  # Otherwise, if it is from the message bus directly, it should be a request
  messageType: event                                 
#  Below is optional configurations settings for mqtt
#  type: mqtt                                               
#  optional:                                                
#    ClientId: client1
#    Username: user1                                        
#    Password: password            
#    Qos: 1                        
#    KeepAlive: 5000           
#    Retained: true/false                                   
#    ConnectionPayload:
#    CertFile:                                              
#    KeyFile:                  
#    CertPEMBlock:                 
#    KeyPEMBlock:              
#    SkipCertVerify: true/false                             
                           
device_conf:                                                
  topic: edgex/events/#/Random-UnsignedInteger-Device/Uint16
  messageType: request

Contents of /etc/kuiper/connections/connection.yaml:

mqtt:                             
  localConnection: #connection key
    server: "tcp://127.0.0.1:1883"
    username: ekuiper                                  
    password: password                              
    #certificationPath: /var/kuiper/xyz-certificate.pem
    #privateKeyPath: /var/kuiper/xyz-private.pem.key
    #rootCaPath: /var/kuiper/xyz-rootca.pem
    #insecureSkipVerify: false    
    #protocolVersion: 3                
  cloudConnection: #connection key
    server: "tcp://broker.emqx.io:1883"
    username: user1                                    
    password: password                             
    #certificationPath: /var/kuiper/xyz-certificate.pem
    #privateKeyPath: /var/kuiper/xyz-private.pem.ke
    #rootCaPath: /var/kuiper/xyz-rootca.pem
    #insecureSkipVerify: false
    #protocolVersion: 3
      
                                    
edgex:             
  redisMsgBus: #redis connection key
    protocol: redis
    server: 127.0.0.1
    port: 6379                                           
    type: redis  
    #  Below is optional configurations settings for mqtt
    #  type: mqtt         
    #  optional:        
    #    ClientId: client1 
    #    Username: user1
    #    Password: password
    #    Qos: 1              
    #    KeepAlive: 5000   
    #    Retained: true/false
    #    ConnectionPayload:
    #    CertFile:    
    #    KeyFile:    
    #    CertPEMBlock:             
    #    KeyPEMBlock:        
    #    SkipCertVerify: true/false
  mqttMsgBus: #connection key
    protocol: tcp
    server: 127.0.0.1
    port: 1883
    type: mqtt       
    optional:
      KeepAlive: "50"        
                 
  zeroMsgBus: #connection key
    protocol: tcp
    server: 10.168.1.1           
    port: 5571

Script used for running the kuiper server:

set -e

# env settings are populated from env files of docker-compose

echo "Script for waiting on security bootstrapping ready-to-run"

# gating on the ready-to-run port
echo "$(date) Executing waitFor with $@ waiting on tcp://${STAGEGATE_BOOTSTRAPPER_HOST}:${STAGEGATE_READY_TORUNPORT}"
./security-bootstrapper --confdir=./res waitFor \
  -uri tcp://"${STAGEGATE_BOOTSTRAPPER_HOST}":"${STAGEGATE_READY_TORUNPORT}" \
  -timeout "${STAGEGATE_WAITFOR_TIMEOUT}"

echo "$(date) Starting Kuiper ..."
#exec   /usr/bin/docker-entrypoint.sh ./bin/kuiperd
#exec   ./entrypoint.sh ./bin/kuiperd
exec    kuiperd

Commands for creating kuiper stream and rule:

kuiper create stream strm'(Uint16 bigint) WITH (FORMAT="JSON", TYPE="edgex", CONF_KEY="device_conf")'

kuiper create rule rule_strm -f rule_demo.txt

Contents of the sql file(rule_demo.txt):

{
"id": "rule_strm",
"sql": "SELECT * FROM strm WHERE Uint16>0",
"actions":[
        {
                "mqtt":
                {
                "server":"tcp://10.168.1.1:1883",
                "topic":"kuiper_status",
                "clientId":"app_002",
                "sendSingle": true,
                "dataTemplate": "{{.Uint16}}"
                }
        }

]
}

[MonicaisHer]

You could change the log level to 'debug' to obtain more information about ekuiper's status.

[MonicaisHer]

You could manually define it to test if it works. There is a manual way to obtain Redis credentials for kuiper by using a Vault token to access the Vault:

$TOKEN=$(yq .auth.client_token ekuiper/secrets-token.json)
curl --header "X-Vault-Token: $TOKEN" \
--request GET http://localhost:8200/v1/secret/edgex/edgex-kuiper/redisdb

The output should contain .data.username and .data.password, which are Redis credentials.

Once you obtain the Redis credentials, you can add them to your edgex.yaml and connection.yaml files as follows:

optional:
  Username: default
  Password: ***

Take a look at this script I used to get the Redis credentials for kuiper in the Snap deployment. Just keep in mind that the path may be a little different for the Docker deployment.

[neerajasjawali]

Unable to generate tokens

root@NB2-SOC-BSP-20FEB23-REL12-v1:~ curl --header "X-Vault-Token: $TOKEN" \--request GET http://myhost:8200/v1/secret/edgex/edgex-kuiper/redisdb
{"errors":["1 error occurred:\n\t* internal error\n\n"]}

[bnevis-i]

I'll provide a further update after this mornings EdgeX TSC meetings.... we don't currently create a secret store token for ekuiper. Need to force this to occur, ADD_SECRETSTORE_TOKENS, ADD_KNOWN_SECRETS.

[bnevis-i]

Update: I added the following to security-secretstore-setup:

      ADD_SECRETSTORE_TOKENS: ekuiper
      ADD_KNOWN_SECRETS: "redisdb[ekuiper]"

And to rules-engine (for the moment, not injecting the bootstrapper script):

  rules-engine:
    read_only: false
    env_file:
      - common-security.env
      - common-sec-stage-gate.env
    volumes:
      - edgex-init:/edgex-init:ro
      - /tmp/edgex/secrets/ekuiper:/tmp/edgex/secrets/ekuiper:ro,z
    depends_on:
      - security-bootstrapper

Add curl

docker exec -ti -u root edgex-kuiper apk add curl

Get secret store token (note workaround for different uid for ekuiper container)

docker exec -ti -u 2002 edgex-kuiper cat /tmp/edgex/secrets/ekuiper/secrets-token.json
{"auth":{"accessor":"LVMoiADPNchxjZsmdiSiYiLF","client_token":"hvs.CAESIOCSe0_...redacted...TzM","entity_id":"62ed024c-ef9a-68c1-b4e9-6cabb7159f9b","identity_policies":["edgex-service-ekuiper"],"lease_duration":3600,"metadata":{"username":"ekuiper"},"mfa_requirement":null,"num_uses":0,"orphan":true,"policies":["default","edgex-service-ekuiper"],"renewable":true,"token_policies":["default"],"token_type":"service"},"data":null,"lease_duration":0,"lease_id":"","renewable":false,"request_id":"fb8ee324-9752-0626-be00-c507585d8cf1","warnings":null,"wrap_info":null}

docker exec -ti -u 2002 edgex-kuiper sh -l
edgex-kuiper:/kuiper$ export VAULT_TOKEN=hvs.CAESIOCSe0_..redacted...TzM
edgex-kuiper:/kuiper$ curl --header "X-Vault-Token: $VAULT_TOKEN" --request GET http://edgex-vault:8200/v1/secret/edgex/ekuiper/redisdb
{"request_id":"99a6aea6-30c4-7c06-9fdf-99ee0002f897","lease_id":"","renewable":false,"lease_duration":604800,"data":{"password":"OpzGZ5Eocn...redacted...Avh","username":"default"},"wrap_info":null,"warnings":null,"auth":null}

So it all works, but it is easier to grab it out of the config files

cat /kuiper/etc/connections/connection.yaml 

edgex:
  redisMsgBus: #connection key
    protocol: redis
    server: localhost
    port: 6379
    type: redis
    optional:
      Username: default
      Password: OpzGZ5Eocn..redacted...Avh

So I can't reproduce your issue, and the assets you need the connection are right there in plain sight.

[neerajasjawali]

Update: I added the following to security-secretstore-setup:

      ADD_SECRETSTORE_TOKENS: ekuiper
      ADD_KNOWN_SECRETS: "redisdb[ekuiper]"

And to rules-engine (for the moment, not injecting the bootstrapper script):

  rules-engine:
    read_only: false
    env_file:
      - common-security.env
      - common-sec-stage-gate.env
    volumes:
      - edgex-init:/edgex-init:ro
      - /tmp/edgex/secrets/ekuiper:/tmp/edgex/secrets/ekuiper:ro,z
    depends_on:
      - security-bootstrapper

Add curl

docker exec -ti -u root edgex-kuiper apk add curl

Get secret store token (note workaround for different uid for ekuiper container)

docker exec -ti -u 2002 edgex-kuiper cat /tmp/edgex/secrets/ekuiper/secrets-token.json
{"auth":{"accessor":"LVMoiADPNchxjZsmdiSiYiLF","client_token":"hvs.CAESIOCSe0_...redacted...TzM","entity_id":"62ed024c-ef9a-68c1-b4e9-6cabb7159f9b","identity_policies":["edgex-service-ekuiper"],"lease_duration":3600,"metadata":{"username":"ekuiper"},"mfa_requirement":null,"num_uses":0,"orphan":true,"policies":["default","edgex-service-ekuiper"],"renewable":true,"token_policies":["default"],"token_type":"service"},"data":null,"lease_duration":0,"lease_id":"","renewable":false,"request_id":"fb8ee324-9752-0626-be00-c507585d8cf1","warnings":null,"wrap_info":null}

docker exec -ti -u 2002 edgex-kuiper sh -l
edgex-kuiper:/kuiper$ export VAULT_TOKEN=hvs.CAESIOCSe0_..redacted...TzM
edgex-kuiper:/kuiper$ curl --header "X-Vault-Token: $VAULT_TOKEN" --request GET http://edgex-vault:8200/v1/secret/edgex/ekuiper/redisdb
{"request_id":"99a6aea6-30c4-7c06-9fdf-99ee0002f897","lease_id":"","renewable":false,"lease_duration":604800,"data":{"password":"OpzGZ5Eocn...redacted...Avh","username":"default"},"wrap_info":null,"warnings":null,"auth":null}

So it all works, but it is easier to grab it out of the config files

cat /kuiper/etc/connections/connection.yaml 

edgex:
  redisMsgBus: #connection key
    protocol: redis
    server: localhost
    port: 6379
    type: redis
    optional:
      Username: default
      Password: OpzGZ5Eocn..redacted...Avh

So I can't reproduce your issue, and the assets you need the connection are right there in plain sight.

Thank you @bnevis-i for helping me out. I would try testing it accordingly and get back to you. Thanks again

[neerajasjawali]

@bnevis-i What is the significance of the env_files and where can I find them?

rules-engine:
    read_only: false
    env_file:
      - common-security.env
      - common-sec-stage-gate.env
    volumes:
      - edgex-init:/edgex-init:ro
      - /tmp/edgex/secrets/ekuiper:/tmp/edgex/secrets/ekuiper:ro,z
    depends_on:
      - security-bootstrapper

[neerajasjawali]

You could manually define it to test if it works. There is a manual way to obtain Redis credentials for kuiper by using a Vault token to access the Vault:

$TOKEN=$(yq .auth.client_token ekuiper/secrets-token.json)
curl --header "X-Vault-Token: $TOKEN" \
--request GET http://localhost:8200/v1/secret/edgex/edgex-kuiper/redisdb

The output should contain .data.username and .data.password, which are Redis credentials.

Once you obtain the Redis credentials, you can add them to your edgex.yaml and connection.yaml files as follows:

optional:
  Username: default
  Password: ***

Take a look at this script I used to get the Redis credentials for kuiper in the Snap deployment. Just keep in mind that the path may be a little different for the Docker deployment.

@MonicaisHer Thank you for your suggestions.

[bnevis-i]

@bnevis-i What is the significance of the env_files and where can I find them?

rules-engine:
    read_only: false
    env_file:
      - common-security.env
      - common-sec-stage-gate.env
    volumes:
      - edgex-init:/edgex-init:ro
      - /tmp/edgex/secrets/ekuiper:/tmp/edgex/secrets/ekuiper:ro,z
    depends_on:
      - security-bootstrapper

Example is taken from here:

https://github.com/edgexfoundry/edgex-compose/blob/main/compose-builder/add-security.yml#L341

What is missing from above is the

   entrypoint: ["/edgex-init/ready_to_run_wait_install.sh"]
   command: ["/usr/bin/docker-entrypoint.sh", "./bin/kuiperd"]

(which waits for the secretstore to initialize and stuff) and an appropriate command to launch eKuiper (which is a concatenation of entrypoint and cmd for the ekuiper docker image)

[bnevis-i]

This discussion has gone quite a long way, and I am going to throw this in, but not sure it is relevant, but:

security-secretstore-setup won't generate an secure MQTT credential unless the SecureMesageBus.Type setting is mqtt. The docker-compose files can be generated with an option to set the env var SECUREMESSAGEBUS_TYPE=mqtt.

This setting causes a message-bus secret to be added to the secret store, as well as appropriate eKuiper configuration files to be generated.

The security-enabled default is SECUREMESSAGEBUS_TYPE=redis which should do the same for redis as the message bus.

[bnevis-i]

Here is my edgex.yaml

application_conf:
  port: 5571
  protocol: tcp
  server: localhost
  topic: application
default:
  optional:
    Username: default
    Password: 9e2...redacted....yIh
  port: 6379
  protocol: redis
  server: localhost
  connectionSelector: edgex.redisMsgBus
  topic: rules-events
  type: redis
mqtt_conf:
  optional:
    ClientId: client1
  port: 1883
  protocol: tcp
  server: localhost
  topic: events
  type: mqtt

and my connection.yaml

edgex:
  redisMsgBus: #connection key
    protocol: redis
    server: localhost
    port: 6379
    type: redis
    optional:
      Username: default
      Password: 9e2...redacted...yIh

[neerajasjawali]

When the redis is running in "protected-mode no", Kuiper is able to communicate with Redis and get the data from the messagebus. As for my application I am running the redis is secure mode. So, how can I make the kuiper to connect to the redis when in secure mode?

[lenny-goodell]

Note that eKuiper does run with EdgeX in Secure Mode. In jakarta, the Redis Pub/Sub is secure and EdgeX injects the MessageBus credentials for Redis Pub/Sub. MQTT MessageBus in Jakarta is not secured and eKuiper doesn't need credentials to connect.

The Levski release added secure MQTT MessageBus and EdgeX injects the MessageBus credentials for the MQTT Broker.

[neerajasjawali]

Note that eKuiper does run with EdgeX in Secure Mode. In jakarta, the Redis Pub/Sub is secure and EdgeX injects the MessageBus credentials for Redis Pub/Sub. MQTT MessageBus in Jakarta is not secured and eKuiper doesn't need credentials to connect.

The Levski release added secure MQTT MessageBus and EdgeX injects the MessageBus credentials for the MQTT Broker.

@lenny-intel Redis Pub/Sub is secure means that its authenticated by username & password or when its running in protected-mode?

[bnevis-i]

As I updated in response https://github.com/orgs/edgexfoundry/discussions/85#discussioncomment-5598749, the connections you need to connect to redis directly are there, or could be made available relatively easy. Please make sure you are running a current release of EdgeX.

[neerajasjawali]

As I updated in response https://github.com/orgs/edgexfoundry/discussions/85#discussioncomment-5598749, the connections you need to connect to redis directly are there, or could be made available relatively easy. Please make sure you are running a current release of EdgeX.

@bnevis-i I am currently working with jakarta LTS release. Do I still need to update the version?

[neerajasjawali]

@ngjaying @bnevis-i @lenny-intel Thank you so much for your help.
I am able to get data on the kuiper end. As suggested in (https://github.com/orgs/edgexfoundry/discussions/85#discussioncomment-5598749) by @bnevis-i, I have added the kuiper to security-secretstore-setup and updated the connection.yaml file respectively. Its working as expected. Thanks again.

[bnevis-i]

Great. See my amendment here: https://github.com/orgs/edgexfoundry/discussions/85#discussioncomment-5605459 that will make sure that kuiper initialization is held off until the secret store is initialized.