[BUG] Dashboard fails to connect when using securityconfig

[jonathon2nd]

Describe the bug

Dashboard does not connect when trying to set and use different passwords.

To Reproduce
Steps to reproduce the behavior:
Deploy cluster from operator.

securityconfig-secret is taken from https://github.com/Opster/opensearch-k8s-operator/blob/main/opensearch-operator/examples/opensearch-cluster-securityconfig.yaml
I have rehashed passwords by using the hash.sh I found on the opensearch pod, as getting it working locally failed.

admin-credentials-secret

---
apiVersion: v1
kind: Secret
metadata:
  name: admin-credentials-secret
  namespace: labs-opensearch
type: Opaque
stringData:
  username: admin
  password: PASSWORD

Values for cluster

---
apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: labs-cluster
  namespace: labs-opensearch
spec:
  security:
    tls:  # Everything related to TLS configuration
      transport:  # Configuration of the transport endpoint
        generate: true  # Have the operator generate and sign certificates
        perNode: true  # Separate certificate per node
      http:  # Configuration of the HTTP endpoint
        generate: true  # Have the Operator generate and sign certificates
    config:
      securityConfigSecret:
        name: securityconfig-secret
      adminCredentialsSecret:
        name: admin-credentials-secret
  general:
    serviceName: labs-cluster
    version: 1.3.1
    setVMMaxMapCount: true
  dashboards:
    enable: true
    additionalConfig:
      opensearch_security.auth.type: "proxy"
      opensearch.requestHeadersWhitelist: |
        ["securitytenant","Authorization","x-forwarded-for","x-auth-request-access-token", "x-auth-request-email", "x-auth-request-groups"]
      opensearch_security.multitenancy.enabled: "true"
      opensearch.username: "admin"
      opensearch.password: "PASSWORD"
    tls:
      enable: true  # Configure TLS
      generate: true  # Have the Operator generate and sign a certificate
    version: 1.3.1
    replicas: 1
    resources:
      requests:
         memory: "512Mi"
         cpu: "200m"
      limits:
         memory: "512Mi"
         cpu: "200m"
  nodePools:
    - component: masters
      replicas: 3
      diskSize: "50Gi"
      jvm: -Xmx1024M -Xms1024M #Set to half of memory request
      persistence:
        pvc:
          storageClass: linstor-replica-one-local
          accessModes:
          - ReadWriteOnce
      NodeSelector:
      resources:
         requests:
            memory: "2Gi"
            cpu: "500m"
         limits:
            memory: "2Gi"
            cpu: "500m"
      roles:
        - "data"
        - "master"

Expected behavior
Expect dashboard to connect, like when testing without changing passwords

OpenSearch Version
Please list the version of OpenSearch being used.
Operator: 2.1.1
general.version: 1.3.1

Dashboards Version
Please list the version of OpenSearch Dashboards being used.
dashboards.version: 1.3.1

Plugins

Please list all plugins currently enabled.
Default

Screenshots

If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: k8s
• Browser and version: chrome Version 107.0.5304.87 (Official Build) (64-bit)

Additional context

Add any other context about the problem here.
Following quite the rabbit hole trying to get this working. I have been following https://github.com/Opster/opensearch-k8s-operator/blob/main/docs/userguide/main.md as well as I can, 404'd links aside.

I have consulted https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/config/opensearch_dashboards.yml and have tried a number of different dashboards.additionalConfig settings, including empty, and I get the same results. I have defaulted to trying with the admin user, as you can see in provided yaml.

Looked at opensearch-project/security#1787, opensearch-project/security#1836, opensearch-project/security#1576, opensearch-project/opensearch-k8s-operator#305 hoping to find some clues, so far no.

[jonathon2nd]

Changing to 1.3.6 for both opensearch and dashboard does not change anything.
Re-validated that install works without trying to modify security.

[jonathon2nd]

OK I got a working setup with https://github.com/Opster/opensearch-k8s-operator/blob/main/opensearch-operator/examples/opensearch-cluster-securityconfig.yaml and https://github.com/Opster/opensearch-k8s-operator/blob/main/opensearch-operator/examples/securityconfig-secret.yaml
So I must have some issue with my config I tried to build from the docs, either that or the updated passwords I hashed did not work?

[jonathon2nd]

If I just change the passwords I get the following

[2022-11-01T18:07:14,755][WARN ][o.o.s.a.BackendRegistry ] [my-cluster-masters-0] Authentication finally failed for admin from 127.0.0.1:52994
--
Tue, Nov 1 2022 12:07:44 pm | [2022-11-01T18:07:44,626][WARN ][o.o.s.a.BackendRegistry ] [my-cluster-masters-0] Authentication finally failed for admin from 127.0.0.1:55022
Tue, Nov 1 2022 12:07:46 pm | [2022-11-01T18:07:46,061][WARN ][o.o.s.a.BackendRegistry ] [my-cluster-masters-0] Authentication finally failed for admin from 127.0.0.1:55160
Tue, Nov 1 2022 12:08:14 pm | [2022-11-01T18:08:14,730][WARN ][o.o.s.a.BackendRegistry ] [my-cluster-masters-0] Authentication finally failed for admin from 127.0.0.1:57000

And the dashboard also does not connect, but when adding admin-credentials-secret with the same password,

But the exact same problem is happening, dashboard will not connect

{"type":"log","@timestamp":"2022-11-01T19:27:15Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
--
Tue, Nov 1 2022 1:27:17 pm | {"type":"log","@timestamp":"2022-11-01T19:27:17Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
Tue, Nov 1 2022 1:27:20 pm | {"type":"log","@timestamp":"2022-11-01T19:27:20Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
Tue, Nov 1 2022 1:27:20 pm | {"type":"log","@timestamp":"2022-11-01T19:27:20Z","tags":["info","plugins-system"],"pid":1,"message":"Stopping all plugins."}
Tue, Nov 1 2022 1:27:20 pm | {"type":"log","@timestamp":"2022-11-01T19:27:20Z","tags":["info","savedobjects-service"],"pid":1,"message":"Starting saved objects migrations"}
Tue, Nov 1 2022 1:27:20 pm | {"type":"log","@timestamp":"2022-11-01T19:27:20Z","tags":["warning","savedobjects-service"],"pid":1,"message":"Unable to connect to OpenSearch. Error: Given the configuration, the ConnectionPool was not able to find a usable Connection for this request."}

Redid password hashing, so made sure that it was the same.

[opensearch@my-cluster-masters-0 ~]$ /usr/share/opensearch/plugins/opensearch-security/tools/hash.sh 

[bandinib-amzn]

Issue moved to opensearch-project/security-dashboards-plugin #1177 via Zenhub

[bandinib-amzn]

Moved to opensearch-project/security-dashboards-plugin#1177