How to enable edgex security mode under docker swarm overlay #225

Jan5566 · 2023-12-07T08:04:45Z

Jan5566
Dec 7, 2023

The problem is that edgex can work normally under docker swarm overlay, but can not work properly when enable security mode. Please help me solve this problem thanks.

Edgex version is 3.1.0

my docker-compose.yml
docker-compose.zip

find error in consul log:

edgex_consul.1.rc1swnffcdfd    | level=WARN ts=2023-12-07T07:43:27.0014072Z app=security-bootstrapper source=command.go:572 msg="error from getting Consul leader API call, will retry it again: Failed to send request for http URL: Get \"http://edgex-core-consul:8500/v1/status/leader\": dial tcp 10.0.8.33:8500: connect: connection refused"
edgex_consul.1.rc1swnffcdfd    | level=WARN ts=2023-12-07T07:43:28.001883Z app=security-bootstrapper source=command.go:572 msg="error from getting Consul leader API call, will retry it again: Failed to send request for http URL: Get \"http://edgex-core-consul:8500/v1/status/leader\": dial tcp 10.0.8.33:8500: connect: connection refused"
edgex_consul.1.rc1swnffcdfd   | level=WARN ts=2023-12-07T07:43:29.0023879Z app=security-bootstrapper source=command.go:572 msg="error from getting Consul leader API call, will retry it again: Failed to send request for http URL: Get \"http://edgex-core-consul:8500/v1/status/leader\": dial tcp 10.0.8.33:8500: connect: connection refused"
edgex_consul.1.rc1swnffcdfd    | level=WARN ts=2023-12-07T07:43:30.0029907Z app=security-bootstrapper source=command.go:572 msg="error from getting Consul leader API call, will retry it again: Failed to send request for http URL: Get \"http://edgex-core-consul:8500/v1/status/leader\": dial tcp 10.0.8.33:8500: connect: connection refused"
edgex_consul.1.rc1swnffcdfd    | level=WARN ts=2023-12-07T07:43:31.0034701Z app=security-bootstrapper source=command.go:572 msg="error from getting Consul leader API call, will retry it again: Failed to send request for http URL: Get \"http://edgex-core-consul:8500/v1/status/leader\": dial tcp 10.0.8.33:8500: connect: connection refused"
edgex_consul.1.rc1swnffcdfd    | level=ERROR ts=2023-12-07T07:43:32.0046671Z app=security-bootstrapper source=init.go:90 msg="failed to wait for Consul leader: timed out to get non-empty Consul leader"
edgex_consul.1.rc1swnffcdfd    | Thu Dec  7 07:43:32 UTC 2023 failed to set up Consul ACL
edgex_consul.1.rc1swnffcdfd    | Thu Dec  7 07:43:32 UTC 2023 Changing ownership of consul token path to 2002:2001
edgex_consul.1.rc1swnffcdfd    | chown: /tmp/edgex/secrets/consul-acl-token/mgmt_token.json: No such file or directory
edgex_consul.1.rc1swnffcdfd    | level=INFO ts=2023-12-07T07:43:32.0175008Z app=security-bootstrapper source=config.go:731 msg="Loading configuration file from /edgex-init/res/configuration.yaml"
edgex_consul.1.rc1swnffcdfd    | level=INFO ts=2023-12-07T07:43:32.0179794Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Ready/ToRunPort' by environment variable: STAGEGATE_READY_TORUNPORT=54329"
edgex_consul.1.rc1swnffcdfd    | level=INFO ts=2023-12-07T07:43:32.0180326Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Registry/ACL/BootstrapTokenPath' by environment variable: STAGEGATE_REGISTRY_ACL_BOOTSTRAPTOKENPATH=/tmp/edgex/secrets/consul-acl-token/bootstrap_token.json"
edgex_consul.1.rc1swnffcdfd    | level=INFO ts=2023-12-07T07:43:32.0180455Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/BootStrapper/StartPort' by environment variable: STAGEGATE_BOOTSTRAPPER_STARTPORT=54321"
edgex_consul.1.rc1swnffcdfd    | level=INFO ts=2023-12-07T07:43:32.0180532Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Registry/ReadyPort' by environment variable: STAGEGATE_REGISTRY_READYPORT=54324"
edgex_consul.1.rc1swnffcdfd    | level=INFO ts=2023-12-07T07:43:32.0180602Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Registry/Host' by environment variable: STAGEGATE_REGISTRY_HOST=edgex-core-consul"
edgex_consul.1.rc1swnffcdfd    | level=INFO ts=2023-12-07T07:43:32.0180855Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Registry/Port' by environment variable: STAGEGATE_REGISTRY_PORT=8500"
edgex_consul.1.rc1swnffcdfd    | level=INFO ts=2023-12-07T07:43:32.0181255Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Registry/ACL/ManagementTokenPath' by environment variable: STAGEGATE_REGISTRY_ACL_MANAGEMENTTOKENPATH=/tmp/edgex/secrets/consul-acl-token/mgmt_token.json"
edgex_consul.1.rc1swnffcdfd    | level=INFO ts=2023-12-07T07:43:32.0181394Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'SecretStore/Host' by environment variable: SECRETSTORE_HOST=edgex-vault"
edgex_consul.1.rc1swnffcdfd    | level=INFO ts=2023-12-07T07:43:32.0181722Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/BootStrapper/Host' by environment variable: STAGEGATE_BOOTSTRAPPER_HOST=edgex-security-bootstrapper"
edgex_consul.1.rc1swnffcdfd    | level=INFO ts=2023-12-07T07:43:32.0181869Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Database/ReadyPort' by environment variable: STAGEGATE_DATABASE_READYPORT=6379"
edgex_consul.1.rc1swnffcdfd    | level=INFO ts=2023-12-07T07:43:32.0181989Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Database/Port' by environment variable: STAGEGATE_DATABASE_PORT=6379"
edgex_consul.1.rc1swnffcdfd    | level=INFO ts=2023-12-07T07:43:32.0182098Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/SecretStoreSetup/Host' by environment variable: STAGEGATE_SECRETSTORESETUP_HOST=edgex-security-secretstore-setup"
edgex_consul.1.rc1swnffcdfd    | level=INFO ts=2023-12-07T07:43:32.0182215Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/SecretStoreSetup/Tokens/ReadyPort' by environment variable: STAGEGATE_SECRETSTORESETUP_TOKENS_READYPORT=54322"
edgex_consul.1.rc1swnffcdfd    | level=INFO ts=2023-12-07T07:43:32.0182537Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Database/Host' by environment variable: STAGEGATE_DATABASE_HOST=edgex-redis"
edgex_consul.1.rc1swnffcdfd    | level=INFO ts=2023-12-07T07:43:32.0182833Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Registry/ACL/SentinelFilePath' by environment variable: STAGEGATE_REGISTRY_ACL_SENTINELFILEPATH=/consul/config/consul_acl_done"
edgex_consul.1.rc1swnffcdfd    | level=INFO ts=2023-12-07T07:43:32.0182999Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/WaitFor/Timeout' by environment variable: STAGEGATE_WAITFOR_TIMEOUT=60s"
edgex_consul.1.rc1swnffcdfd    | level=INFO ts=2023-12-07T07:43:32.0183088Z app=security-bootstrapper source=config.go:251 msg="Private configuration loaded from file with 16 overrides applied"
edgex_consul.1.rc1swnffcdfd    | level=INFO ts=2023-12-07T07:43:32.0186023Z app=security-bootstrapper source=command.go:81 msg="Security bootstrapper running listenTcp"
edgex_consul.1.rc1swnffcdfd    | level=INFO ts=2023-12-07T07:43:32.0208309Z app=security-bootstrapper source=listener.go:58 msg="Security bootstrapper starts listening on tcp://edgex-core-consul:54324"
edgex_consul.1.rc1swnffcdfd    | level=INFO ts=2023-12-07T07:43:32.0945836Z app=security-bootstrapper source=listener.go:67 msg="Accepted connection on edgex-core-consul:54324"

Other micro service such as edgex_core-command, edgex_core-data ,edgex_core-metadata, etc. ,failed to get Configuration Provider (consul) access token as following log.

edgex_core-command.1.au5gq3pnvyuq    | Thu Dec  7 15:59:03 CST 2023 Starting /core-command -cp=consul.http://edgex-core-consul:8500 --registry ...
edgex_core-command.1.au5gq3pnvyuq    | level=INFO ts=2023-12-07T07:59:03.8620072Z app=core-command source=secret.go:69 msg="Creating SecretClient"
edgex_core-command.1.au5gq3pnvyuq    | level=INFO ts=2023-12-07T07:59:03.8633932Z app=core-command source=variables.go:462 msg="Variables override of 'SecretStore/Host' by environment variable: SECRETSTORE_HOST=edgex-vault"
edgex_core-command.1.au5gq3pnvyuq    | level=INFO ts=2023-12-07T07:59:03.8638005Z app=core-command source=secret.go:172 msg="SecretStore information created with 1 overrides applied"
edgex_core-command.1.au5gq3pnvyuq    | level=INFO ts=2023-12-07T07:59:03.8644129Z app=core-command source=secret.go:79 msg="Reading secret store configuration and authentication token"
edgex_core-command.1.au5gq3pnvyuq    | level=INFO ts=2023-12-07T07:59:03.8644821Z app=core-command source=secret.go:216 msg="load token from file"
edgex_core-command.1.au5gq3pnvyuq    | level=INFO ts=2023-12-07T07:59:03.8646126Z app=core-command source=secret.go:101 msg="Attempting to create secret client"
edgex_core-command.1.au5gq3pnvyuq    | level=INFO ts=2023-12-07T07:59:03.8675068Z app=core-command source=secret.go:112 msg="Created SecretClient"
edgex_core-command.1.au5gq3pnvyuq    | level=INFO ts=2023-12-07T07:59:03.86756Z app=core-command source=secret.go:117 msg="SecretsFile not set, skipping seeding of service secrets."
edgex_core-command.1.au5gq3pnvyuq    | level=INFO ts=2023-12-07T07:59:03.867605Z app=core-command source=secrets.go:277 msg="kick off token renewal with interval: 30m0s"
edgex_core-command.1.au5gq3pnvyuq    | level=ERROR ts=2023-12-07T07:59:03.8694376Z app=core-command source=bootstrap.go:48 msg="failed to create provider for all-services: failed to get Configuration Provider (consul) access token: HTTP response with status code 400, message: failed to generate Consul token using [core-command]: {\"errors\":[\"role \\\"core-command\\\" not found\"]}\n"

bnevis-i · 2023-12-07T15:16:46Z

bnevis-i
Dec 7, 2023
Maintainer

Jan,

How many nodes in your swarm cluster? Have you tried running it in a one-node swarm cluster first?

Also, we have a Kubernetes helm chart for EdgeX deployment here: https://github.com/edgexfoundry/edgex-helm

The difficulty running EdgeX in secure mode in any kind of multi-node orchestration framework has generally been distributing the secret store token to the EdgeX services on remote nodes -- pretty much you need a network file system like GlusterFS running across your nodes.

Also, I suggest you read Is EdgeX Foundry Cloud Native? that explains the EdgeX position on a lot of these topics.

7 replies

Jan5566 Dec 13, 2023
Author

I want to run it on the manager node first. If that can work fine, I will try it on remote nodes.

bnevis-i Dec 13, 2023
Maintainer

Understood.

Jan5566 Dec 15, 2023
Author

Hi @bnevis-i,

Sorry question again, when I check the edgex-consul log, I found it can not send request for url:"http://edgex-core-consul:8500/v1/status/leader and not found mgmt_token.json. Is this normal?

edgex_consul.1.xreqpfm6mp83    | level=WARN ts=2023-12-15T09:56:16.002164Z app=security-bootstrapper source=command.go:572 msg="error from getting Consul leader API call, will retry it again: Failed to send request for http URL: Get \"http://edgex-core-consul:8500/v1/status/leader\": dial tcp 10.0.33.47:8500: connect: connection refused"
edgex_consul.1.xreqpfm6mp83    | level=ERROR ts=2023-12-15T09:56:17.0026841Z app=security-bootstrapper source=init.go:90 msg="failed to wait for Consul leader: timed out to get non-empty Consul leader"
edgex_consul.1.xreqpfm6mp83    | Fri Dec 15 09:56:17 UTC 2023 failed to set up Consul ACL
edgex_consul.1.xreqpfm6mp83    | Fri Dec 15 09:56:17 UTC 2023 Changing ownership of consul token path to 2002:2001
edgex_consul.1.xreqpfm6mp83    | chown: /tmp/edgex/secrets/consul-acl-token/mgmt_token.json: No such file or directory

I can enter edgex-consul container, but consul command not work.

/ # consul members
Error retrieving members: Get "http://127.0.0.1:8500/v1/agent/members?segment=_all": dial tcp 127.0.0.1:8500: connect: connection refused

Maybe it's consul not work normally that other server can not register to consul and the error as following. Any ideal to fix the situation? Thanks.

edgex_device-virtual.1.phq4nv9vuktq   | level=ERROR ts=2023-12-15T11:59:04.8153299Z app=device-virtual source=bootstrap.go:48 msg="failed to create provider for all-services: failed to get Configuration Provider (consul) access token: HTTP response with status code 400, message: failed to generate Consul token using [device-virtual]: {\"errors\":[\"role \\\"device-virtual\\\" not found\"]}\n"

bnevis-i Dec 15, 2023
Maintainer

I agree with your assessment that Consul is not coming up properly.

mgmt_token.json is set by this code
https://github.com/edgexfoundry/edgex-go/blob/420b6f8ee6b7ced97b7db737aa2a08d41b6b2623/cmd/security-bootstrapper/entrypoint-scripts/consul_wait_install.sh#L76

Which replaces the default Consul entrypoint script here:
https://github.com/edgexfoundry/edgex-compose/blob/main/docker-compose.yml#L112

My suggestion is to replace the consul entrypoint script and command with come commands to just hang. (e.g. tail -f /dev/null). (Also it will help if you make read_only: false so you can write files while debugging.)

Then, run the commands of consul_wait_install.sh by hand in order to figure out why Consul is not starting properly.

Jan5566 Apr 18, 2024
Author

Hi @bnevis-i

I'm now trying the Kubernetes helm chart for EdgeX deployment here. But it seems not work when enable security.

NB:~/edgex-helm$ kubectl get pods -n edgex
NAME                                                     READY   STATUS             RESTARTS        AGE
edgex-app-rules-engine-777776567c-58dc9                  0/1     CrashLoopBackOff   7 (3m45s ago)   15m
edgex-core-command-7cbf4fdcbb-th4vp                      0/1     CrashLoopBackOff   7 (3m41s ago)   15m
edgex-core-common-config-bootstrapper-5b5f6b7dd9-hpb9c   0/1     CrashLoopBackOff   7 (3m31s ago)   15m
edgex-core-consul-787d4d7d4-mh4fk                        1/1     Running            0               15m
edgex-core-data-dbbd887f-z9vc7                           0/1     CrashLoopBackOff   7 (3m38s ago)   15m
edgex-core-metadata-67bb5654c4-nh2xp                     0/1     CrashLoopBackOff   7 (3m15s ago)   15m
edgex-device-rest-74c79b4f88-vxvng                       0/1     CrashLoopBackOff   7 (3m24s ago)   15m
edgex-device-virtual-7d7cf9df78-tqzw9                    0/1     CrashLoopBackOff   7 (3m19s ago)   15m
edgex-kuiper-85864fdf54-lg4f8                            1/1     Running            0               15m
edgex-redis-64df67bdf8-687bs                             1/1     Running            0               15m
edgex-security-bootstrapper-b664748bf-4czmh              1/1     Running            0               15m
edgex-security-secretstore-setup-d9f98bdcd-jx47f         1/1     Running            0               15m
edgex-support-notifications-7b8bd88b49-k7scz             0/1     CrashLoopBackOff   7 (3m18s ago)   15m
edgex-support-scheduler-64687cc7f6-wlh6l                 0/1     CrashLoopBackOff   7 (3m36s ago)   15m
edgex-vault-7df85b66f-cl4q9

edgex-core-consul log:

NB:~/edgex-helm$ kubectl logs edgex-core-consul-787d4d7d4-mh4fk -n edgex
Tue Apr 16 09:28:33 UTC 2024 CONSUL_LOCAL_CONFIG:
{
    "enable_local_script_checks": true,
    "disable_update_check": true,
    "ports": {
      "dns": -1
    }
}

Tue Apr 16 09:28:33 UTC 2024 Starting edgex-core-consul with ACL enabled ...
Tue Apr 16 09:28:33 UTC 2024 Executing waitFor on Consul with waiting on TokensReadyPort   tcp://edgex-security-secretstore-setup:54322
level=INFO ts=2024-04-16T09:28:33.7098044Z app=security-bootstrapper source=config.go:731 msg="Loading configuration file from /edgex-init/res/configuration.yaml"
level=INFO ts=2024-04-16T09:28:33.7115099Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/WaitFor/Timeout' by environment variable: STAGEGATE_WAITFOR_TIMEOUT=60s"
level=INFO ts=2024-04-16T09:28:33.7115816Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/SecretStoreSetup/Host' by environment variable: STAGEGATE_SECRETSTORESETUP_HOST=edgex-security-secretstore-setup"
level=INFO ts=2024-04-16T09:28:33.7116016Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Database/Port' by environment variable: STAGEGATE_DATABASE_PORT=6379"
level=INFO ts=2024-04-16T09:28:33.7116153Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Registry/ReadyPort' by environment variable: STAGEGATE_REGISTRY_READYPORT=54324"
level=INFO ts=2024-04-16T09:28:33.7116313Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Registry/Port' by environment variable: STAGEGATE_REGISTRY_PORT=8500"
level=INFO ts=2024-04-16T09:28:33.7116427Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/BootStrapper/StartPort' by environment variable: STAGEGATE_BOOTSTRAPPER_STARTPORT=54321"
level=INFO ts=2024-04-16T09:28:33.7116535Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Database/Host' by environment variable: STAGEGATE_DATABASE_HOST=edgex-redis"
level=INFO ts=2024-04-16T09:28:33.7116647Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Registry/ACL/SentinelFilePath' by environment variable: STAGEGATE_REGISTRY_ACL_SENTINELFILEPATH=/consul/config/consul_acl_done"
level=INFO ts=2024-04-16T09:28:33.7116929Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Registry/ACL/ManagementTokenPath' by environment variable: STAGEGATE_REGISTRY_ACL_MANAGEMENTTOKENPATH=/tmp/edgex/secrets/consul-acl-token/mgmt_token.json"
level=INFO ts=2024-04-16T09:28:33.7117003Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Database/ReadyPort' by environment variable: STAGEGATE_DATABASE_READYPORT=6379"
level=INFO ts=2024-04-16T09:28:33.7117087Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'SecretStore/Host' by environment variable: SECRETSTORE_HOST=edgex-vault"
level=INFO ts=2024-04-16T09:28:33.7117147Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Registry/Host' by environment variable: STAGEGATE_REGISTRY_HOST=edgex-core-consul"
level=INFO ts=2024-04-16T09:28:33.7117211Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Registry/ACL/BootstrapTokenPath' by environment variable: STAGEGATE_REGISTRY_ACL_BOOTSTRAPTOKENPATH=/tmp/edgex/secrets/consul-acl-token/bootstrap_token.json"
level=INFO ts=2024-04-16T09:28:33.7117301Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/BootStrapper/Host' by environment variable: STAGEGATE_BOOTSTRAPPER_HOST=edgex-security-bootstrapper"
level=INFO ts=2024-04-16T09:28:33.7117411Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/SecretStoreSetup/Tokens/ReadyPort' by environment variable: STAGEGATE_SECRETSTORESETUP_TOKENS_READYPORT=54322"
level=INFO ts=2024-04-16T09:28:33.711752Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Ready/ToRunPort' by environment variable: STAGEGATE_READY_TORUNPORT=54329"
level=INFO ts=2024-04-16T09:28:33.7117643Z app=security-bootstrapper source=config.go:251 msg="Private configuration loaded from file with 16 overrides applied"
level=INFO ts=2024-04-16T09:28:33.7121608Z app=security-bootstrapper source=command.go:119 msg="Security bootstrapper running waitFor"
level=INFO ts=2024-04-16T09:28:33.7122825Z app=security-bootstrapper source=command.go:144 msg="Waiting for: [tcp://edgex-security-secretstore-setup:54322] with timeout: [1m0s]"
==> Starting Consul agent...
               Version: '1.16.2'
            Build Date: '2023-09-19 19:29:18 +0000 UTC'
               Node ID: 'aae569ee-63b6-3545-95f1-3e3bb3dc772a'
             Node name: 'edgex-core-consul'
            Datacenter: 'dc1' (Segment: '<all>')
                Server: true (Bootstrap: true)
           Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, gRPC: -1, gRPC-TLS: 8503, DNS: -1)
          Cluster Addr: 10.244.2.220 (LAN: 8301, WAN: 8302)
     Gossip Encryption: false
      Auto-Encrypt-TLS: false
           ACL Enabled: true
     Reporting Enabled: false
    ACL Default Policy: deny
             HTTPS TLS: Verify Incoming: false, Verify Outgoing: false, Min Version: TLSv1_2
              gRPC TLS: Verify Incoming: false, Min Version: TLSv1_2
      Internal RPC TLS: Verify Incoming: false, Verify Outgoing: false (Verify Hostname: false), Min Version: TLSv1_2

==> Log data will now stream in as it occurs:

2024-04-16T09:28:33.824Z [WARN]  agent: skipping file /consul/config/consul_acl_done, extension must be .hcl or .json, or config format must be set
2024-04-16T09:28:33.824Z [WARN]  agent: bootstrap = true: do not enable unless necessary
2024-04-16T09:28:33.830Z [WARN]  agent.auto_config: skipping file /consul/config/consul_acl_done, extension must be .hcl or .json, or config format must be set
2024-04-16T09:28:33.830Z [WARN]  agent.auto_config: bootstrap = true: do not enable unless necessary
2024-04-16T09:28:33.866Z [INFO]  agent.server.raft: initial configuration: index=1998 servers="[{Suffrage:Voter ID:aae569ee-63b6-3545-95f1-3e3bb3dc772a Address:10.244.2.194:8300}]"
2024-04-16T09:28:33.867Z [INFO]  agent.server.raft: entering follower state: follower="Node at 10.244.2.220:8300 [Follower]" leader-address= leader-id=
2024-04-16T09:28:33.867Z [INFO]  agent.server.serf.wan: serf: EventMemberJoin: edgex-core-consul.dc1 10.244.2.220
2024-04-16T09:28:33.879Z [INFO]  agent.server.serf.lan: serf: EventMemberJoin: edgex-core-consul 10.244.2.220
2024-04-16T09:28:33.879Z [INFO]  agent.router: Initializing LAN area manager
2024-04-16T09:28:33.879Z [WARN]  agent.server.serf.wan: serf: Failed to re-join any previously known node
2024-04-16T09:28:33.879Z [WARN]  agent.server.serf.lan: serf: Failed to re-join any previously known node
2024-04-16T09:28:33.879Z [INFO]  agent.server: Handled event for server in area: event=member-join server=edgex-core-consul.dc1 area=wan
2024-04-16T09:28:33.879Z [INFO]  agent.server: Adding LAN server: server="edgex-core-consul (Addr: tcp/10.244.2.220:8300) (DC: dc1)"
2024-04-16T09:28:33.880Z [INFO]  agent.server.autopilot: reconciliation now disabled
2024-04-16T09:28:33.881Z [INFO]  agent.server.cert-manager: initialized server certificate management
2024-04-16T09:28:33.885Z [INFO]  agent: Starting server: address=[::]:8500 network=tcp protocol=http
2024-04-16T09:28:33.885Z [INFO]  agent: Started gRPC listeners: port_name=grpc_tls address=[::]:8503 network=tcp
2024-04-16T09:28:33.886Z [INFO]  agent: started state syncer
2024-04-16T09:28:33.886Z [INFO]  agent: Consul agent running!
2024-04-16T09:28:35.184Z [WARN]  agent: Check is now critical: check=device-virtual
2024-04-16T09:28:35.525Z [WARN]  agent: Check is now critical: check=core-data
2024-04-16T09:28:36.124Z [WARN]  agent: Check is now critical: check=support-notifications
2024-04-16T09:28:38.392Z [WARN]  agent: Check is now critical: check=core-command
2024-04-16T09:28:39.107Z [WARN]  agent: Check is now critical: check=device-rest
2024-04-16T09:28:40.700Z [WARN]  agent.server.raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
2024-04-16T09:28:40.700Z [INFO]  agent.server.raft: entering candidate state: node="Node at 10.244.2.220:8300 [Candidate]" term=43
2024-04-16T09:28:40.715Z [INFO]  agent.server.raft: election won: term=43 tally=1
2024-04-16T09:28:40.715Z [INFO]  agent.server.raft: entering leader state: leader="Node at 10.244.2.220:8300 [Leader]"
2024-04-16T09:28:40.715Z [INFO]  agent.server: cluster leadership acquired
2024-04-16T09:28:40.715Z [INFO]  agent.server: New leader elected: payload=edgex-core-consul
2024-04-16T09:28:40.839Z [INFO]  agent.server: initializing acls
2024-04-16T09:28:40.845Z [INFO]  agent.leader: started routine: routine="acl token reaping"
2024-04-16T09:28:40.845Z [INFO]  agent.server.autopilot: reconciliation now enabled
2024-04-16T09:28:40.845Z [INFO]  agent.leader: started routine: routine="federation state anti-entropy"
2024-04-16T09:28:40.845Z [INFO]  agent.leader: started routine: routine="federation state pruning"
2024-04-16T09:28:40.845Z [INFO]  agent.leader: started routine: routine="streaming peering resources"
2024-04-16T09:28:40.845Z [INFO]  agent.leader: started routine: routine="metrics for streaming peering resources"
2024-04-16T09:28:40.845Z [INFO]  agent.leader: started routine: routine="peering deferred deletion"
2024-04-16T09:28:40.845Z [INFO]  connect.ca: initialized primary datacenter CA from existing CARoot with provider: provider=consul
2024-04-16T09:28:40.845Z [INFO]  agent.leader: started routine: routine="intermediate cert renew watch"
2024-04-16T09:28:40.845Z [INFO]  agent.leader: started routine: routine="CA root pruning"
2024-04-16T09:28:40.845Z [INFO]  agent.leader: started routine: routine="CA root expiration metric"
2024-04-16T09:28:40.845Z [INFO]  agent.leader: started routine: routine="CA signing expiration metric"
2024-04-16T09:28:40.845Z [INFO]  agent.leader: started routine: routine="virtual IP version check"
2024-04-16T09:28:40.845Z [INFO]  agent.leader: started routine: routine="config entry controllers"
2024-04-16T09:28:40.845Z [INFO]  agent.leader: stopping routine: routine="virtual IP version check"
2024-04-16T09:28:40.845Z [INFO]  agent.leader: stopped routine: routine="virtual IP version check"
2024-04-16T09:28:40.845Z [INFO]  agent.server.raft: updating configuration: command=AddVoter server-id=aae569ee-63b6-3545-95f1-3e3bb3dc772a server-addr=10.244.2.220:8300 servers="[{Suffrage:Voter ID:aae569ee-63b6-3545-95f1-3e3bb3dc772a Address:10.244.2.220:8300}]"
2024-04-16T09:28:40.851Z [INFO]  agent.server: member joined, marking health alive: member=edgex-core-consul partition=default
2024-04-16T09:28:42.431Z [WARN]  agent: Check is now critical: check=core-metadata
level=INFO ts=2024-04-16T09:28:42.7200393Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-secretstore-setup on 10.96.0.10:53: server misbehaving. Sleeping 1s"
2024-04-16T09:28:43.205Z [WARN]  agent: Check is now critical: check=app-rules-engine
2024-04-16T09:28:43.884Z [WARN]  agent: Check is now critical: check=support-scheduler
2024-04-16T09:28:44.382Z [WARN]  agent: Node info update blocked by ACLs: node=aae569ee-63b6-3545-95f1-3e3bb3dc772a accessorID="anonymous token"
2024-04-16T09:28:44.382Z [WARN]  agent: Service registration blocked by ACLs: service=core-metadata accessorID="anonymous token"
2024-04-16T09:28:44.382Z [WARN]  agent: Service registration blocked by ACLs: service=support-scheduler accessorID="anonymous token"
2024-04-16T09:28:44.382Z [WARN]  agent: Service registration blocked by ACLs: service=device-rest accessorID="anonymous token"
2024-04-16T09:28:44.382Z [WARN]  agent: Service registration blocked by ACLs: service=app-rules-engine accessorID="anonymous token"
2024-04-16T09:28:44.382Z [WARN]  agent: Service registration blocked by ACLs: service=core-data accessorID="anonymous token"
2024-04-16T09:28:44.382Z [WARN]  agent: Service registration blocked by ACLs: service=device-virtual accessorID="anonymous token"
2024-04-16T09:28:44.382Z [WARN]  agent: Service registration blocked by ACLs: service=core-command accessorID="anonymous token"
2024-04-16T09:28:44.382Z [WARN]  agent: Service registration blocked by ACLs: service=support-notifications accessorID="anonymous token"
2024-04-16T09:28:45.186Z [WARN]  agent: Check is now critical: check=device-virtual
2024-04-16T09:28:45.579Z [WARN]  agent: Check is now critical: check=core-data
2024-04-16T09:28:46.126Z [WARN]  agent: Check is now critical: check=support-notifications
2024-04-16T09:28:46.456Z [WARN]  agent: Node info update blocked by ACLs: node=aae569ee-63b6-3545-95f1-3e3bb3dc772a accessorID="anonymous token"
2024-04-16T09:28:46.456Z [WARN]  agent: Service registration blocked by ACLs: service=core-command accessorID="anonymous token"
2024-04-16T09:28:46.456Z [WARN]  agent: Service registration blocked by ACLs: service=support-notifications accessorID="anonymous token"
2024-04-16T09:28:46.456Z [WARN]  agent: Service registration blocked by ACLs: service=core-metadata accessorID="anonymous token"
2024-04-16T09:28:46.456Z [WARN]  agent: Service registration blocked by ACLs: service=support-scheduler accessorID="anonymous token"
2024-04-16T09:28:46.456Z [WARN]  agent: Service registration blocked by ACLs: service=device-rest accessorID="anonymous token"
2024-04-16T09:28:46.456Z [WARN]  agent: Service registration blocked by ACLs: service=app-rules-engine accessorID="anonymous token"

edgex-device-virtual log:

NB:~/edgex-helm$ kubectl logs -f edgex-device-virtual-7d7cf9df78-tqzw9 -n edgex
Script for waiting on security bootstrapping ready-to-run
Tue Apr 16 09:50:27 UTC 2024 Executing waitFor with /device-virtual -cp=consul.http://edgex-core-consul:8500 --registry waiting on tcp://edgex-security-bootstrapper:54329
level=INFO ts=2024-04-16T09:50:27.6414144Z app=security-bootstrapper source=config.go:731 msg="Loading configuration file from /edgex-init/res/configuration.yaml"
level=INFO ts=2024-04-16T09:50:27.6423545Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Ready/ToRunPort' by environment variable: STAGEGATE_READY_TORUNPORT=54329"
level=INFO ts=2024-04-16T09:50:27.6424208Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Registry/ReadyPort' by environment variable: STAGEGATE_REGISTRY_READYPORT=54324"
level=INFO ts=2024-04-16T09:50:27.6424389Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'SecretStore/Host' by environment variable: SECRETSTORE_HOST=edgex-vault"
level=INFO ts=2024-04-16T09:50:27.6424517Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Database/Port' by environment variable: STAGEGATE_DATABASE_PORT=6379"
level=INFO ts=2024-04-16T09:50:27.6424711Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/SecretStoreSetup/Host' by environment variable: STAGEGATE_SECRETSTORESETUP_HOST=edgex-security-secretstore-setup"
level=INFO ts=2024-04-16T09:50:27.6440344Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Database/Host' by environment variable: STAGEGATE_DATABASE_HOST=edgex-redis"
level=INFO ts=2024-04-16T09:50:27.6447354Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Database/ReadyPort' by environment variable: STAGEGATE_DATABASE_READYPORT=6379"
level=INFO ts=2024-04-16T09:50:27.6448086Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Registry/Port' by environment variable: STAGEGATE_REGISTRY_PORT=8500"
level=INFO ts=2024-04-16T09:50:27.6448454Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/BootStrapper/Host' by environment variable: STAGEGATE_BOOTSTRAPPER_HOST=edgex-security-bootstrapper"
level=INFO ts=2024-04-16T09:50:27.6448839Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/SecretStoreSetup/Tokens/ReadyPort' by environment variable: STAGEGATE_SECRETSTORESETUP_TOKENS_READYPORT=54322"
level=INFO ts=2024-04-16T09:50:27.6449108Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/WaitFor/Timeout' by environment variable: STAGEGATE_WAITFOR_TIMEOUT=60s"
level=INFO ts=2024-04-16T09:50:27.6449516Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/BootStrapper/StartPort' by environment variable: STAGEGATE_BOOTSTRAPPER_STARTPORT=54321"
level=INFO ts=2024-04-16T09:50:27.6449802Z app=security-bootstrapper source=variables.go:462 msg="Variables override of 'StageGate/Registry/Host' by environment variable: STAGEGATE_REGISTRY_HOST=edgex-core-consul"
level=INFO ts=2024-04-16T09:50:27.6450223Z app=security-bootstrapper source=config.go:251 msg="Private configuration loaded from file with 13 overrides applied"
level=INFO ts=2024-04-16T09:50:27.6455048Z app=security-bootstrapper source=command.go:119 msg="Security bootstrapper running waitFor"
level=INFO ts=2024-04-16T09:50:27.6456101Z app=security-bootstrapper source=command.go:144 msg="Waiting for: [tcp://edgex-security-bootstrapper:54329] with timeout: [1m0s]"
level=INFO ts=2024-04-16T09:50:27.6472061Z app=security-bootstrapper source=command.go:214 msg="Connected to tcp://edgex-security-bootstrapper:54329"
Tue Apr 16 09:50:27 UTC 2024 Starting /device-virtual -cp=consul.http://edgex-core-consul:8500 --registry ...
level=INFO ts=2024-04-16T09:50:27.666503Z app=device-virtual source=secret.go:69 msg="Creating SecretClient"
level=INFO ts=2024-04-16T09:50:27.6667997Z app=device-virtual source=variables.go:462 msg="Variables override of 'SecretStore/Host' by environment variable: SECRETSTORE_HOST=edgex-vault"
level=INFO ts=2024-04-16T09:50:27.6669236Z app=device-virtual source=secret.go:172 msg="SecretStore information created with 1 overrides applied"
level=INFO ts=2024-04-16T09:50:27.6669452Z app=device-virtual source=secret.go:79 msg="Reading secret store configuration and authentication token"
level=INFO ts=2024-04-16T09:50:27.6669669Z app=device-virtual source=secret.go:216 msg="load token from file"
level=INFO ts=2024-04-16T09:50:27.6671087Z app=device-virtual source=secret.go:101 msg="Attempting to create secret client"
level=INFO ts=2024-04-16T09:50:27.6730341Z app=device-virtual source=secret.go:112 msg="Created SecretClient"
level=INFO ts=2024-04-16T09:50:27.673099Z app=device-virtual source=secret.go:117 msg="SecretsFile not set, skipping seeding of service secrets."
level=INFO ts=2024-04-16T09:50:27.6733573Z app=device-virtual source=secrets.go:277 msg="kick off token renewal with interval: 30m0s"
level=ERROR ts=2024-04-16T09:50:27.6822331Z app=device-virtual source=bootstrap.go:48 msg="failed to create provider for all-services: failed to get Configuration Provider (consul) access token: HTTP response with status code 400, message: failed to generate Consul token using [device-virtual]: {\"errors\":[\"Unexpected response code: 403 (ACL system must be bootstrapped before making any requests that require authorization: ACL not found)\"]}\n"

bnevis-i · 2024-04-18T14:50:40Z

bnevis-i
Apr 18, 2024
Maintainer

@Jan5566 I'm not at a system where I can run a quick regression on this, but the following line looks suspicious:

level=INFO ts=2024-04-16T09:28:42.7200393Z app=security-bootstrapper source=command.go:209 msg="Problem with dial: dial tcp: lookup edgex-security-secretstore-setup on 10.96.0.10:53: server misbehaving. Sleeping 1s"

I'm not immediately sure what's wrong, but this is the tell:

level=ERROR ts=2024-04-16T09:50:27.6822331Z app=device-virtual source=bootstrap.go:48 msg="failed to create provider for all-services: failed to get Configuration Provider (consul) access token: HTTP response with status code 400, message: failed to generate Consul token using [device-virtual]: {\"errors\":[\"Unexpected response code: 403 (ACL system must be bootstrapped before making any requests that require authorization: ACL not found)\"]}\n"

So something during the Consul initialization is causing Consul to not be switched into ACL mode. This is going to fail every other service that depends on Consul.

That all happens in this script: https://github.com/edgexfoundry/edgex-go/blob/main/cmd/security-bootstrapper/entrypoint-scripts/consul_wait_install.sh

It looks a lot like the problem earlier in the thread, repeating itself in Kubernetes.

0 replies

bnevis-i · 2024-04-19T01:38:35Z

bnevis-i
Apr 19, 2024
Maintainer

I pulled from the main branch of edgex-helm and clicked it off in secure mode using hostPath storage. It took about 4 minutes from a completely cold system to get everything loaded and running. The bottleneck to get it running was edgex-vault and security-secretstore-setup finishing their thing, since due to their size they seem to be the last to start, causing all the others to fail and retry. While waiting, I observed "i/o timeout" instead of "server misbehaving". But eventually, it all came up in less than 5 minutes.

In the hostPath configuration, all the services will start on a single node. I suggest going to the /mnt folder on that node and completely wiping any edgex data folders there and starting again. The documentation specifically notes you can't switch in and out of secure mode without a full clean.

I think the repo is fine and any problems are with your configuration, I'm sorry to say.

2 replies

Jan5566 Apr 25, 2024
Author

@bnevis-i It ran successfully after I deleted the folder. But the another question is how to deploy a specific pod to a specific node in security mode. Is it also need NFS to distribute the secret store token to the EdgeX services on remote nodes

I have two nodes, one is master and the other is worker. But it seem always run on worker in security mode.

NAME                                                     READY   STATUS    RESTARTS      AGE   IP           NODE         NOMINATED NODE   READINESS GATES
edgex-security-bootstrapper-b664748bf-tplmk              1/1     Running   0             50s   10.42.1.13   k8s-worker   <none>           <none>
edgex-core-common-config-bootstrapper-78c898c964-zgpw5   1/1     Running   0             47s   10.42.1.24   k8s-worker   <none>           <none>
edgex-core-consul-d87548cfc-v6nbq                        1/1     Running   0             50s   10.42.1.26   k8s-worker   <none>           <none>
edgex-vault-5ccdf9fd8d-82728                             1/1     Running   0             51s   10.42.1.14   k8s-worker   <none>           <none>
edgex-security-secretstore-setup-6697dc74b9-45xdr        1/1     Running   0             47s   10.42.1.21   k8s-worker   <none>           <none>
edgex-redis-5fbdc86789-t4799                             1/1     Running   0             48s   10.42.1.19   k8s-worker   <none>           <none>
edgex-kuiper-74fc4b98d5-8f75v                            1/1     Running   0             48s   10.42.1.27   k8s-worker   <none>           <none>
edgex-core-command-7b49d596df-9s4zw                      1/1     Running   0             50s   10.42.1.22   k8s-worker   <none>           <none>
edgex-support-scheduler-c84d784c9-n6vgp                  1/1     Running   0             50s   10.42.1.25   k8s-worker   <none>           <none>
edgex-core-data-6b7c9fb4f4-8t4dm                         1/1     Running   0             48s   10.42.1.16   k8s-worker   <none>           <none>
edgex-core-metadata-6558bdd789-6fbcx                     1/1     Running   0             49s   10.42.1.18   k8s-worker   <none>           <none>
edgex-support-notifications-55f5dd4c96-bp9rn             1/1     Running   0             49s   10.42.1.20   k8s-worker   <none>           <none>
edgex-app-rules-engine-6786c4cf6d-w8bgr                  1/1     Running   0             50s   10.42.1.15   k8s-worker   <none>           <none>
edgex-device-rest-6ff9576695-44w9k                       1/1     Running   1 (23s ago)   50s   10.42.1.17   k8s-worker   <none>           <none>
edgex-device-virtual-84bfc74d5f-j8dvx                    1/1     Running   2 (23s ago)   49s   10.42.1.23   k8s-worker   <none>           <none>

All pods are pending after I set the edgex.security.enabled flag to true.

NAME                                                     READY   STATUS    RESTARTS   AGE     IP       NODE     NOMINATED NODE   READINESS GATES
edgex-app-rules-engine-565dc8f6b4-545gj                  0/1     Pending   0          8m58s   <none>   <none>   <none>           <none>
edgex-core-common-config-bootstrapper-78f4d4d448-2bcqk   0/1     Pending   0          8m58s   <none>   <none>   <none>           <none>
edgex-core-consul-bdb475dbc-dg2md                        0/1     Pending   0          8m57s   <none>   <none>   <none>           <none>
edgex-core-command-8574ff8c79-2ng5p                      0/1     Pending   0          8m57s   <none>   <none>   <none>           <none>
edgex-core-data-79f88b566b-spdrf                         0/1     Pending   0          8m57s   <none>   <none>   <none>           <none>
edgex-device-rest-6bb56776d9-t2wbf                       0/1     Pending   0          8m56s   <none>   <none>   <none>           <none>
edgex-device-virtual-758b99499c-797wg                    0/1     Pending   0          8m55s   <none>   <none>   <none>           <none>
edgex-kuiper-6f97b887fc-qwh8d                            0/1     Pending   0          8m55s   <none>   <none>   <none>           <none>
edgex-core-metadata-6c77bd99d7-b5ht7                     0/1     Pending   0          8m55s   <none>   <none>   <none>           <none>
edgex-redis-799cbc784c-khq2b                             0/1     Pending   0          8m54s   <none>   <none>   <none>           <none>
edgex-security-bootstrapper-b664748bf-kt94h              0/1     Pending   0          8m54s   <none>   <none>   <none>           <none>
edgex-support-notifications-669b69b496-nbhtc             0/1     Pending   0          8m54s   <none>   <none>   <none>           <none>
edgex-support-scheduler-765f8767df-2hmjt                 0/1     Pending   0          8m53s   <none>   <none>   <none>           <none>
edgex-vault-5b9b6546f6-mfvcp                             0/1     Pending   0          8m53s   <none>   <none>   <none>           <none>
edgex-security-secretstore-setup-6bfc69c865-rnlx9        0/1     Pending   0          8m54s   <none>   <none>   <none>           <none>

bnevis-i Apr 29, 2024
Maintainer

I have two nodes, one is master and the other is worker. But it seem always run on worker in security mode.

https://github.com/edgexfoundry/edgex-helm/blob/main/values.yaml#L347

That doesn't seem right. security.enabled defaults to false and that should block creation of edgex-vault, edgex-security-secretstore-setup, and edgex-security-bootstrapper out of the box.

@bnevis-i It ran successfully after I deleted the folder. But the another question is how to deploy a specific pod to a specific node in security mode. Is it also need NFS to distribute the secret store token to the EdgeX services on remote nodes.

Correct. The default values.yaml is set up so that if you --set storage.useHostPath=false then it will create shared volume (rook-cephfs storageClass by default) to distribute the Vault tokens. This should be sufficient, when security is disabled, for most of the services to distribute. (https://github.com/edgexfoundry/edgex-helm/blob/main/values.yaml#L649-L659) When security is enabled, both the Vault tokens and the edgex-init scritpts (for container entrypoint overrides) need to land on the shared volume.

But the another question is how to deploy a specific pod to a specific node in security mode.

You'll have to customize the helm chart resources to add specific node placement constraints to the service you want on a specific node. The out-of-the-box configuration has pod affinity rules set up such that if the distribution requirements are not met, secretstore-setup is on the same host as security-bootstrapper, and most of the rest of the pods land on the same node as secretstore-setup.

I am not sure why you are having trouble. The placement logic is pretty easy to read (https://github.com/edgexfoundry/edgex-helm/blob/main/templates/edgex-support-notifications/edgex-support-notifications-deployment.yaml#L23-L31).

I presume you have already removed the scheduling taint for the master node in your k8s deployment? If you haven't, by default no workload, EdgeX or not, can land on k8s control plane nodes.

kubectl taint nodes $(hostname) node-role.kubernetes.io/master:NoSchedule-

Jan5566 · 2024-05-07T07:58:38Z

Jan5566
May 7, 2024
Author

Correct. The default values.yaml is set up so that if you --set storage.useHostPath=false then it will create shared volume (rook-cephfs storageClass by default) to distribute the Vault tokens. This should be sufficient, when security is disabled, for most of the services to distribute. (https://github.com/edgexfoundry/edgex-helm/blob/main/values.yaml#L649-L659) When security is enabled, both the Vault tokens and the edgex-init scritpts (for container entrypoint overrides) need to land on the shared volume.

If I enable security and --set storage.useHostPath=false, I also need to deploy rook myself like this or network file system like GlusterFS right? Is that Swarm and Kubernetes require network file system to distribute Vault tokens?

1 reply

bnevis-i May 7, 2024
Maintainer

If I enable security and --set storage.useHostPath=false, I also need to deploy rook myself like this or network file system like GlusterFS right? Is that Swarm and Kubernetes require network file system to distribute Vault tokens?

Yes. It is possible that someone would contribute some code that could import Vault tokens as Kubernetes secrets and distribute them via the secrets mechanism. That would be relatively simple because it could just run at the tail of security-secretstore-setup, but nobody has done that. It should probably be pretty straightforward with kubectl, a shell script, an injected service account token, and some RBAC rules, and adjustments to the helm chart to pass the token via secrets volume.

Going the CSI route, GlusterFS is not a good choice as in-tree support was removed in Kubernetes 1.26 and all of the GlusterFS CSI drivers on github have been deprecated. Rook is heavyweight, but will work. A lighter weight option would be OpenEBS with running a local cluster NFS service. I've not gone this route but it is a documented thing.

If you had an external NFS cluster, then just creating an NFS storage class that uses the existing solution would be the most expedient route. This is similar to the GlusterFS route, except that it lacks the fault tolerance you would get with OpenEBS and an in-cluster NFS, or what you used to get with GlsuterFS.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EdgeX Foundry Project

How to enable edgex security mode under docker swarm overlay #225

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments 10 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

EdgeX Foundry Project

How to enable edgex security mode under docker swarm overlay #225

Jan5566 Dec 7, 2023

Replies: 4 comments · 10 replies

bnevis-i Dec 7, 2023 Maintainer

Jan5566 Dec 13, 2023 Author

bnevis-i Dec 13, 2023 Maintainer

Jan5566 Dec 15, 2023 Author

bnevis-i Dec 15, 2023 Maintainer

Jan5566 Apr 18, 2024 Author

bnevis-i Apr 18, 2024 Maintainer

bnevis-i Apr 19, 2024 Maintainer

Jan5566 Apr 25, 2024 Author

bnevis-i Apr 29, 2024 Maintainer

Jan5566 May 7, 2024 Author

bnevis-i May 7, 2024 Maintainer

Jan5566
Dec 7, 2023

Replies: 4 comments 10 replies

bnevis-i
Dec 7, 2023
Maintainer

Jan5566 Dec 13, 2023
Author

bnevis-i Dec 13, 2023
Maintainer

Jan5566 Dec 15, 2023
Author

bnevis-i Dec 15, 2023
Maintainer

Jan5566 Apr 18, 2024
Author

bnevis-i
Apr 18, 2024
Maintainer

bnevis-i
Apr 19, 2024
Maintainer

Jan5566 Apr 25, 2024
Author

bnevis-i Apr 29, 2024
Maintainer

Jan5566
May 7, 2024
Author

bnevis-i May 7, 2024
Maintainer