[BUG] Unable to set OPENSEARCH_INITIAL_ADMIN_PASSWORD for bootstrap pod

[danielkubat]

What is the bug?

Unable to perform new deployment of opensearch 2.12.0 using opensearch-operator 2.5.1.

How can one reproduce the bug?

Create clean opensearch 2.12.0 cluster using opensearch-operator 2.5.1.

What is the expected behavior?

Cluster is up and running including bootstrap pod.

What is your host/environment?

GKE (1.27.8-gke.1067004)

Do you have any additional context?

Opensearch 2.12.0 explicitely requires to set OPENSEARCH_INITIAL_ADMIN_PASSWORD environment variable for each pod. This is possible for nodePools but not for bootstrap pod, as the helm manifest only supports additionalConfig, therefore pod is crashing.

Setting up Custom Admin User does not help, as pod strictly requires OPENSEARCH_INITIAL_ADMIN_PASSWORD defined as the variable.

[ebenezar-mccoy]

I think that once/if #754 is merged, this problem will be solved.

[prudhvigodithi]

[Triage]
Hey @danielkubat and @ebenezar-mccoy, based on my testing in past #703 (comment), change this with Custom Admin User should update the admin user password, else should remain to password as admin.

Thank you

Adding @bbarani @salyh @jochenkressin @pchmielnik @bbarani

[nijave]

I tried to create a cluster with the following config but it's not working with the same error

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: generate-secrets
  namespace: opensearch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: create-secret-role
  namespace: opensearch
rules:
- verbs: ["create", "get"]
  apiGroups: [""]
  resources: ["secrets"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: generate-secrets-create-secret
  namespace: opensearch
subjects:
- kind: ServiceAccount
  name: generate-secrets
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: create-secret-role
---
apiVersion: batch/v1
kind: Job
metadata:
  name: generate-secrets
  namespace: opensearch
spec:
  template:
    spec:
      serviceAccountName: generate-secrets
      securityContext:
        runAsUser: 0
        runAsGroup: 0
      containers:
      - name: kubectl
        image: docker.io/bitnami/kubectl:1.29
        command: [/bin/sh, -c]
        args:
        - |
          apt update
          apt install -y python3 python3-bcrypt

          PASSWORD=$(tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 40)
          PASSWORD_HASH=$(cat <<EOF | python3
          import bcrypt
          print(
            bcrypt.hashpw(
              "$PASSWORD".encode("utf-8"),
              bcrypt.gensalt(12, prefix=b"2a"),
            ).decode("utf-8")
          )
          EOF
          )

          cat <<EOF | kubectl create -f -
          apiVersion: v1
          kind: Secret
          type: Opaque
          metadata:
            name: admin-credentials
            namespace: opensearch
          data:
            username: $(printf "admin" | base64 -w 0)
            password: $(printf "$PASSWORD" | base64 -w 0)
          EOF

          cat <<EOF | kubectl create -f -
          apiVersion: v1
          kind: Secret
          type: Opaque
          metadata:
            name: security-config
            namespace: opensearch
          stringData:
              internal_users.yml: |-
                _meta:
                  type: "internalusers"
                  config_version: 2
                admin:
                  hash: $PASSWORD_HASH
                  reserved: true
                  backend_roles:
                  - "admin"
                  description: "Admin user"
          EOF

          exit 0
      restartPolicy: OnFailure
---
apiVersion: opensearch.opster.io/v1
kind: OpenSearchCluster
metadata:
  name: default-cluster
  namespace: opensearch
spec:
  general:
    version: "2.14.0"
    httpPort: 9200
    vendor: opensearch
    serviceName: opensearch
    monitoring:
     enable: true
    pluginsList: ["repository-s3"]
    setVMMaxMapCount: true
  security:
    config:
      adminCredentialsSecret:
        name: admin-credentials
      securityConfigSecret:
        name: security-config
  dashboards:
    version: "2.14.0"
    enable: true
    replicas: 1
    resources:
      requests:
         memory: 256Mi
         cpu: 50m
      limits:
         memory: 1Gi
         cpu: 500m
  confMgmt:
    smartScaler: true
  nodePools:
    - component: masters
      replicas: 3
      diskSize: 4Gi
      nodeSelector:
      resources:
         requests:
            memory: 512Mi
            cpu: 50m
         limits:
            memory: 768Mi
            cpu: 250m
      roles:
        - master
        - ingest
    - component: nodes
      replicas: 3
      diskSize: 40Gi
      nodeSelector:
      resources:
         requests:
            memory: 1Gi
            cpu: 100m
         limits:
            memory: 2Gi
            cpu: 1000m
      roles:
        - data

This should run a Job to follow steps in Custom Admin User but bootstrap still crashes with No custom admin password found. Please provide a password via the environment variable OPENSEARCH_INITIAL_ADMIN_PASSWORD.

[nijave]

It works adding these env vars. The operator can add this var referencing the Secret its stored in with ValueFrom. I'll put up a PR tonight after work if I have time

[nijave]

I opened #816. It may need some input/review from contributors/maintainers and additional changes.

[nijave]

Couldn't get it working locally without this but the current code consistency works in CI so there must be something else missing. I'm thinking something is triggering the security plugin & security demo script some times

[nijave]

Hmm I can't seem to repro on Github Actions even with the config that was causing me errors locally. Maybe there's some kind of race condition or something at play

[Wolfeg]

probably you can add

bootstrap:
  additionalConfig:
    OPENSEARCH_INITIAL_ADMIN_PASSWORD: 'password'

and it should work, cause operator merge this values to pod env

[nijave]

probably you can add

bootstrap:
  additionalConfig:
    OPENSEARCH_INITIAL_ADMIN_PASSWORD: 'password'

and it should work, cause operator merge this values to pod env

I think it's still missing from one of the pods--maybe bootstrap pod

Edit: Yeah it looks like that's static env vars for bootstrap pod https://github.com/opensearch-project/opensearch-k8s-operator/blob/main/opensearch-operator/pkg/builders/cluster.go#L470

That was one of the things I adjusted in the PR (make sure all pods use common env vars)

[joelp172]

Also experiencing this issue when trying to create a 2.13.0 cluster. Do you have any ideas when a fix will be released?

[kstiehl]

Hi hi,
I am looking at this issues and the issues that are linked here and I've to say the current state is not obvious for everyone.

Is someone able to explain the current state.
From my point of view it seems that it is currently not possible to securely bootstrap a cluster.
Securly means that the admin credentials are only pulled from a secret and not from spec.bootstrap.additionalConfig (since it doesn't allow valueFrom). The PR #754 mentioned by @ebenezar-mccoy is also not changing that.

The PR #816 from @nijave looks promising in that regard. So is there anything we can do here to support that PR being merged?

And in the meantime? Has anyone an Idea on how to bootstrap a cluster ?

[danielkubat]

And in the meantime? Has anyone an Idea on how to bootstrap a cluster ?

yeah, Install 2.11.0 and then upgrade to a version you want to. during upgrade, bootstrap pod is no longer used.

[nijave]

The PR #816 from @nijave looks promising in that regard. So is there anything we can do here to support that PR being merged?

My PR was meant to address an issue where the Opensearch security plugin seemingly gets activated but the operator doesn't correctly setup/bootstrap pods with an initial admin password that meets complexity requirements enforced by the security plugin.

I don't completely understand how the security plugin works but it looks like this is largely out of scope of the operator and built-in to the Opensearch Docker container. I think installing the s3 repository plugin pulled in the Opensearch security plugin causing bootstrap to fail but I don't have a very good understanding of how this is supposed to work and couldn't reliably reproduce it.

There are some docs somewhere for bootstrapping a cluster with your own credentials in the operator docs

It seems like the operator should do something here to play nice with Opensearch security but I don't have enough knowledge and time to figure out the correct approach.

[Blarc]

I noticed, that as soon as I remove security, this error occurs. Edit: with version 2.14.0.

  security:
    tls:
      http:
        generate: true
      transport:
        generate: true
        perNode: true

[prudhvigodithi]

Hey the Operator does not run install_demo_configuration.sh which requires OPENSEARCH_INITIAL_ADMIN_PASSWORD starting from 2.12 version. You can use this link for the admin password update https://github.com/opensearch-project/opensearch-k8s-operator/blob/main/docs/userguide/main.md#custom-admin-user.

I have added some details to this GitHub issue #703 (comment) for 2.12.0 version, please check.

Also from the same comment #703 (comment) the default should be admin:admin right (since the operator does not use the install_demo_configuration.sh) ? @nijave

Also if the cluster already has the security index created, upgrading the cluster (>=2.12.0) does not impact the admin password or user as the security index already has the data.

For new clusters the default is still admin:admin (from #703 (comment)) To update the password we add the securityConfigSecret and adminSecret, the cluster should be up and running with the password part of adminSecret right.

Cluster spec

spec:
  security:
    config: 
     securityConfigSecret:
        name: securityconfig-secret
     adminCredentialsSecret:
        name: admin-credentials-secret

adminCredentialsSecret

apiVersion: v1
kind: Secret
metadata:  
  name: admin-credentials-secret
type: Opaque
data:
  # admin
  username: YWRtaW4=
  # test
  password: dGVzdA==

securityConfigSecret

apiVersion: v1
kind: Secret
metadata:
  name: securityconfig-secret
type: Opaque
## admin opassword hash for test "$2y$12$B6GMBQIwOUEV2qtBQrpJL.37MUMp1XkLxCyWzeTH5Q94QxNjw8ng6"
stringData:
      action_groups.yml: |-
         _meta:
           type: "actiongroups"
           config_version: 2
      internal_users.yml: |-
        _meta:
          type: "internalusers"
          config_version: 2
        admin:
          hash: "$2y$12$tS0wrbNssQpVjOXDPrzqdO5phJC/Fmb9fNKSdJ9P2voGK.LNIqLxG"
          reserved: true
          backend_roles:
          - "admin"
          description: "Demo admin user"
        dashboarduser:
          hash: "$2a$12$4AcgAt3xwOWadA5s5blL6ev39OXDNhmOesEoo33eZtrq2N0YrU3H."
          reserved: true
          description: "Demo OpenSearch Dashboards user"
      nodes_dn.yml: |-
        _meta:
          type: "nodesdn"
          config_version: 2
      whitelist.yml: |-
        _meta:
          type: "whitelist"
          config_version: 2
      tenants.yml: |-
        _meta:
          type: "tenants"
          config_version: 2
      roles_mapping.yml: |-
        _meta:
          type: "rolesmapping"
          config_version: 2
        all_access:
          reserved: false
          backend_roles:
          - "admin"
          description: "Maps admin to all_access"
        own_index:
          reserved: false
          users:
          - "*"
          description: "Allow full access to an index named like the username"
        readall:
          reserved: false
          backend_roles:
          - "readall"
        manage_snapshots:
          reserved: false
          backend_roles:
          - "snapshotrestore"
        dashboard_server:
          reserved: true
          users:
          - "dashboarduser"
      roles.yml: |-
        _meta:
          type: "roles"
          config_version: 2
        dashboard_read_only:
          reserved: true
        security_rest_api_access:
          reserved: true
        # Allows users to view monitors, destinations and alerts
        alerting_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/alerting/alerts/get'
            - 'cluster:admin/opendistro/alerting/destination/get'
            - 'cluster:admin/opendistro/alerting/monitor/get'
            - 'cluster:admin/opendistro/alerting/monitor/search'
        # Allows users to view and acknowledge alerts
        alerting_ack_alerts:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/alerting/alerts/*'
        # Allows users to use all alerting functionality
        alerting_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster_monitor'
            - 'cluster:admin/opendistro/alerting/*'
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices_monitor'
                - 'indices:admin/aliases/get'
                - 'indices:admin/mappings/get'
        # Allow users to read Anomaly Detection detectors and results
        anomaly_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/ad/detector/info'
            - 'cluster:admin/opendistro/ad/detector/search'
            - 'cluster:admin/opendistro/ad/detectors/get'
            - 'cluster:admin/opendistro/ad/result/search'
            - 'cluster:admin/opendistro/ad/tasks/search'
            - 'cluster:admin/opendistro/ad/detector/validate'
            - 'cluster:admin/opendistro/ad/result/topAnomalies'
        # Allows users to use all Anomaly Detection functionality
        anomaly_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster_monitor'
            - 'cluster:admin/opendistro/ad/*'
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices_monitor'
                - 'indices:admin/aliases/get'
                - 'indices:admin/mappings/get'
        # Allows users to read Notebooks
        notebooks_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/notebooks/list'
            - 'cluster:admin/opendistro/notebooks/get'
        # Allows users to all Notebooks functionality
        notebooks_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/notebooks/create'
            - 'cluster:admin/opendistro/notebooks/update'
            - 'cluster:admin/opendistro/notebooks/delete'
            - 'cluster:admin/opendistro/notebooks/get'
            - 'cluster:admin/opendistro/notebooks/list'
        # Allows users to read observability objects
        observability_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opensearch/observability/get'
        # Allows users to all Observability functionality
        observability_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opensearch/observability/create'
            - 'cluster:admin/opensearch/observability/update'
            - 'cluster:admin/opensearch/observability/delete'
            - 'cluster:admin/opensearch/observability/get'
        # Allows users to read and download Reports
        reports_instances_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/reports/instance/list'
            - 'cluster:admin/opendistro/reports/instance/get'
            - 'cluster:admin/opendistro/reports/menu/download'
        # Allows users to read and download Reports and Report-definitions
        reports_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/reports/definition/get'
            - 'cluster:admin/opendistro/reports/definition/list'
            - 'cluster:admin/opendistro/reports/instance/list'
            - 'cluster:admin/opendistro/reports/instance/get'
            - 'cluster:admin/opendistro/reports/menu/download'
        # Allows users to all Reports functionality
        reports_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/reports/definition/create'
            - 'cluster:admin/opendistro/reports/definition/update'
            - 'cluster:admin/opendistro/reports/definition/on_demand'
            - 'cluster:admin/opendistro/reports/definition/delete'
            - 'cluster:admin/opendistro/reports/definition/get'
            - 'cluster:admin/opendistro/reports/definition/list'
            - 'cluster:admin/opendistro/reports/instance/list'
            - 'cluster:admin/opendistro/reports/instance/get'
            - 'cluster:admin/opendistro/reports/menu/download'
        # Allows users to use all asynchronous-search functionality
        asynchronous_search_full_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/asynchronous_search/*'
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices:data/read/search*'
        # Allows users to read stored asynchronous-search results
        asynchronous_search_read_access:
          reserved: true
          cluster_permissions:
            - 'cluster:admin/opendistro/asynchronous_search/get'
        # Allows user to use all index_management actions - ism policies, rollups, transforms
        index_management_full_access:
          reserved: true
          cluster_permissions:
            - "cluster:admin/opendistro/ism/*"
            - "cluster:admin/opendistro/rollup/*"
            - "cluster:admin/opendistro/transform/*"
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - 'indices:admin/opensearch/ism/*'
        # Allows users to use all cross cluster replication functionality at leader cluster
        cross_cluster_replication_leader_full_access:
          reserved: true
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - "indices:admin/plugins/replication/index/setup/validate"
                - "indices:data/read/plugins/replication/changes"
                - "indices:data/read/plugins/replication/file_chunk"
        # Allows users to use all cross cluster replication functionality at follower cluster
        cross_cluster_replication_follower_full_access:
          reserved: true
          cluster_permissions:
            - "cluster:admin/plugins/replication/autofollow/update"
          index_permissions:
            - index_patterns:
                - '*'
              allowed_actions:
                - "indices:admin/plugins/replication/index/setup/validate"
                - "indices:data/write/plugins/replication/changes"
                - "indices:admin/plugins/replication/index/start"
                - "indices:admin/plugins/replication/index/pause"
                - "indices:admin/plugins/replication/index/resume"
                - "indices:admin/plugins/replication/index/stop"
                - "indices:admin/plugins/replication/index/update"
                - "indices:admin/plugins/replication/index/status_check"
      config.yml: |-
        _meta:
          type: "config"
          config_version: "2"
        config:
          dynamic:
            http:
              anonymous_auth_enabled: false
            authc:
              basic_internal_auth_domain:
                http_enabled: true
                transport_enabled: true
                order: "4"
                http_authenticator:
                  type: basic
                  challenge: true
                authentication_backend:
                  type: intern

Thank you
@getsaurabh02 @swoehrl-mw

[chalut01]

+1