Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security] Demo configuration script requires admin password #4094

Closed
2 tasks done
RyanL1997 opened this issue Oct 2, 2023 · 26 comments
Closed
2 tasks done

[Security] Demo configuration script requires admin password #4094

RyanL1997 opened this issue Oct 2, 2023 · 26 comments
Assignees
Labels
enhancement New Enhancement

Comments

@RyanL1997
Copy link
Contributor

RyanL1997 commented Oct 2, 2023

Is your feature request related to a problem? Please describe

According to the changes in security plugin of security demo configuration script: opensearch-project/security#3329. This change introduces an alternative to the default credentials admin:admin the security plugin currently uses.
In this implementation, a new setting is added to the opensearch.yml file. This setting, plugins.security.authcz.admin.password is parsed on plugin start up and then propagated into the internal user store of each node when they launch During the set up of the security demo config, it will ask user to provide an initial admin password. User can either provide that by defining an environment variable 'initialAdminPassword' with a password string, or create a file 'initialAdminPassword.txt' with a single line that contains the password string and place it under the config folder. The absent of this 'initialAdminPassword' will lead to the failure of the demo config script.

For adapting this change, we need to change the build script to handle the setup for admin password.

Describe the solutions you'd like

We have a talk with @RyanL1997 @DarshitChanpura @peterzhuamazon on this for a bit:

Solution A: Adding a parameter for running the demo config script to set the default admin password as 'admin' (sample implementation in security):

We can add a parameter to security demo configuration setup script, which if not provide this parameter, user will be prompted to provide a secure admin password, and if user provides this parameters which means user decides to skip the setup of admin password, then we’ll proceed with "admin" as default admin password and tell user that they acknowledge to accept an insecure password for this non-prod env.

We believe this change is justifiable given that the primary intent of the demo configuration script is for non-production environments. Our documentation clearly underscores our recommendation against using this demo config for actual production setups. In line with this, introducing this parameter remains consistent with the script's intended use, emphasizing that it is not suitable for production deployments.

Solution B: Users have to be prompted to enter an initial admin password during the set up of security demo configuration script (sample implementation in security):

In the current design, the default internal_user.yml contains the default admin:admin credential inside of it (source code). According to @scrawfor99's original PR, it will only replace the default admin hash with the hash of new password. However, since the failure of demo configuration setup will not fail the initiation of the cluster, the admin credential will remain the same if there is a absent of provided admin password by user during the setup of demo configuration script.

So instead of replace the default admin hash, we are removing the default admin credential from internal_user.yml. This solution introduces the method that if we failed to set up a admin credential during the set up of demo configuration script, it will lead to an absent of admin credential section in internal_users.yml, so that the initiation of the cluster will be failed.

Additional context

Original Change in security repo: opensearch-project/security#3329
Discussions on the original issue: opensearch-project/security#3285

Issues

@RyanL1997 RyanL1997 added enhancement New Enhancement untriaged Issues that have not yet been triaged labels Oct 2, 2023
@RyanL1997
Copy link
Contributor Author

RyanL1997 commented Oct 2, 2023

cc: @DarshitChanpura, @peterzhuamazon, @davidlago, @bbarani, @peternied, @cwperks and @scrawfor99

@peterzhuamazon
Copy link
Member

cc: @CEHENKLE @rishabh6788

@peterzhuamazon
Copy link
Member

peterzhuamazon commented Oct 2, 2023

Need more discussion on this as the solution proposed above would requires build to add the parameter to all the currently releasing distributions.

Security also needs to have documentation on this potential breaking change from their side.
On build side, user should see minimum effect after the change. Thanks.

@DarshitChanpura
Copy link
Member

@peterzhuamazon shared his perspective from build side and I share the sentiment. Here is an approach that allows us to gradually bring this change in :

  1. We allow passing an additional inline argument with initialAdminPassword
  2. If the argument is not present, we then prompt user during installation script to provide an admin password.
  3. (For now) If they choose to not provide password we default it to admin with a warning message in next release this option will be removed, and in next release we remove it. (For 2.11 it might be too close to make this call especially with CI builds and RC generation)

@RyanL1997
Copy link
Contributor Author

To be more clear, the requirement is htat we should not ship with the default admin:admin, and this can only be used in the testing environment.

@RyanL1997
Copy link
Contributor Author

RyanL1997 commented Oct 3, 2023

Hey guys:

I've done some digging into the recent changes. Although we've incorporated modifications into both the main and 2.x branches of our security repository, the outcomes aren't aligning with our expectations.

Our initial goal was for both the demo configuration and the cluster initiation to fail when an initial admin password wasn't supplied. To achieve this, we implemented a strict failure mechanism in our demo configuration script that triggers when the initial admin password is absent (source code). However, from what @peterzhuamazon shared, the build team's script doesn't strictly enforce failure when the demo configuration script encounters an error. This means that even if the demo setup is unsuccessful, the cluster initiation still proceeds.

Given this workflow, our current approach in the demo configuration—to substitute the default hash of "admin:admin" in config/opensearch-security/internal_users.yml with the hash of the manually provided password (source code) —doesn't seem foolproof. If the demo script doesn't succeed, the cluster still initializes and continues to recognize the default "admin:admin" credentials. This is because the original authentication data in config/opensearch-security/internal_users.yml remains unchanged.

@peterzhuamazon
Copy link
Member

peterzhuamazon commented Oct 3, 2023

Adding some painpoints that we will face when consuming the upcoming changes from security:

  • Assuming the change is keeping the old behavior (e.g. adding --UseDefaultPassword so it will still auto create admin:admin user)

    • Tar: startup scripts (Can be customized by User)
    • Zip: startup scripts (Can be customized by User)
    • RPM: spec files (Cannot be customized by user and would need to run upon installation of pkg)
    • DEB: user script (Cannot be customized by user and would need to run upon installation of pkg)
    • Docker: OneTime Setups (Can only be disabled if user rebuild image with their own dockerfile)
    • Docker: Entrypoint Script (Can be disabled by user through env var which then requires user to manually add their own internal_users.yml or similar)
    • Helm: GO templates (Can be customized by power user in combination with Docker changes above)
    • Ansible: Use Tar script (Need user to re-write playbook sections in order to customize)
    • Note that these scripts are used very often by users as they also contain other plugins changes such as PA / KNN plugins upon startup of the cluster
  • Assuming the behavior is changed that user have to be prompted to enter password, or automatically print out a random password for user on the console log

    • Tar: User would need to enter a password or remember to copy the random one, manageable but cumbersome
    • ZIP: Same as above
    • RPM: User cannot even complete the installation, or user will not notice the password in console most of the time, as RPM was meant to be a one-click install process so that users can quickly start the cluster without manual steps. The installation of the pkg will need to hardcode the actions in postinst script and it is unreasonable to think that user would need to input password just to run yum install or systemctl start commands.
    • DEB: Same as above
    • Docker: OneTime Setups will be broken and needs to tweak the dockerfile
    • Docker: Entrypoint Script will hard fail as it is not able to wait for user input. User would then needs to manually tweak as well as design a environment var upon starting the docker container or the container will exit instantly, defeat the purpose of docker with ease of use and startup without manual steps
    • Helm: Similar to docker
    • Ansible: Not possible unless new playbook sections are getting rewritten
    • Note this would 100% break all the related automations from Users

Thanks.

@RyanL1997
Copy link
Contributor Author

RyanL1997 commented Oct 4, 2023

@peterzhuamazon Thanks for the detail. Yes, let's have a discussion upon these 2 options, the changes of both options in security repo are ready now. (cc: @DarshitChanpura @peternied @bbarani @davidlago ).

@peterzhuamazon
Copy link
Member

peterzhuamazon commented Oct 6, 2023

20231006 RC1:

https://github.com/opensearch-project/security/commits/2.11

As of the making of 2.11.0 RC1, the commit of security plugin with demo configuration changes is still in 2.11 branch.

This code only issue an exit 1 if there is no user specified password, thus cause a few issues in distribution:

  1. Using tar as the example here, the tar startup script does not enforce an exit if subroutine exits with the non-zero return code (as some commands are not always expected to go through by design). The security demo configuration script in commit d995dafb90ef2be7613502844e177c89a2a4951f will only issue an exit 1. This means, even if the security script exit, the tar startup script will keep going and start the OpenSearch Cluster. The above commit also failed to remove the existing admin user in internal_user.yml, causing the user to be automatically created despite the attempts to stop the creation.
  2. When user start the tar distribution, they will proceed to see this msg "Unable to find the admin password for the cluster. Please run 'export initialAdminPassword=<your_password>' or create a file $ADMIN_PASSWORD_FILE with a single line that contains the password." in their log, while still able to use admin:admin user to call APIs. This would cause confusions while showing user the incorrect information.
  3. We are not sure if there is any additional behavior changes with above commit at the moment.
  • Note: the above commit also affect Docker, DEB, RPM, WINDOWS, HELM, ANSIBLE distribution. We are also not sure if their behaviors are similar / same or different, causing an inconsistent experience to the users.

Thanks.

@DarshitChanpura
Copy link
Member

@peterzhuamazon This is a known issue in the script where even though the script fails execution, the admin user is still created and that is because a default admin user is always present in internal_users.yml which is picked up during creating of users. This behavior will be fixed in the next iteration of this issue via: opensearch-project/security#3449, opensearch-project/security#3472, #4107

@rishabh6788
Copy link
Collaborator

@DarshitChanpura
Copy link
Member

@rishabh6788 Yes, that is how the user will be able to pick up new password. As of now there is no other place for them to know the generated password (unless they have provided one).

@rishabh6788
Copy link
Collaborator

How about write it to the configuration file and ask user to read it from there?
Not sure how this works out in Managed service but anyone running open-source OpenSearch should be able to read the file and grab the password.

@DarshitChanpura
Copy link
Member

How about write it to the configuration file and ask user to read it from there?

Can you elaborate this? What type of file are you thinking about?

anyone running open-source OpenSearch should be able to read the file and grab the password.

This is if they have access to the logs.

@rishabh6788
Copy link
Collaborator

$OPENSEARCH_CONF_DIR/initialAdminPassword.txt. Write to this file and let user grab it from here. This is just a suggestion as logging passwords is not best practice.

@DarshitChanpura
Copy link
Member

$OPENSEARCH_CONF_DIR/initialAdminPassword.txt this is one of the ways user can specify the initial admin password. If someone has access to this file, they can still grab the password and access the cluster.

@bbarani
Copy link
Member

bbarani commented Oct 6, 2023

@DarshitChanpura We would like to get your inputs on the below 2 issues with this commit.

1.) User experience would not be great if we display this error message "Unable to find the admin password for the cluster. Please run 'export initialAdminPassword=<your_password>' or create a file $ADMIN_PASSWORD_FILE with a single line that contains the password." even though they can still use admin:admin user to call APIs.

2.) Printing out the user / password in log is going to create a non-optimal experience for users who are concerned with security.

We are good to proceed if you think that the above 2 issues doesn't warrant a revert.

We will post any other findings as and when we see it to this thread.

@DarshitChanpura
Copy link
Member

DarshitChanpura commented Oct 6, 2023

Sure thing.

  1. We have captured this in this issue: [BUG] install_demo_configuration script shows confusing log security#3489
  2. The original design was proposed in: https://github.com/OpenSearch-Security/OpenSearch-Security/issues/3 and the solution was implemented keeping this design in mind.

@peternied Would you like to add your thoughts here since you and @scrawfor99 were driving the original change?

@DarshitChanpura
Copy link
Member

DarshitChanpura commented Oct 6, 2023

I have created a PR to address the confusing log: opensearch-project/security#3490. It points to 2.11 and should help resolve outstanding concerns.

@peterzhuamazon
Copy link
Member

peterzhuamazon commented Oct 9, 2023

The latest RC on 2.11.0 as of 20231007 could not startup deb/rpm correctly.
cc: @bbarani @rishabh6788

The pkg will proceed to run the demo configuration script and failed due to no password:

**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
OpenSearch Security Demo Installer
 ** Warning: Do not use on production or public reachable systems **
Basedir: /usr/share/opensearch
This script maybe require your root password for 'sudo' privileges
OpenSearch install type: rpm/deb on NAME="Amazon Linux"
OpenSearch config dir: /etc/opensearch
OpenSearch config file: /etc/opensearch/opensearch.yml
OpenSearch bin dir: /usr/share/opensearch/bin
OpenSearch plugins dir: /usr/share/opensearch/plugins
OpenSearch lib dir: /usr/share/opensearch/lib
Detected OpenSearch Version: x-content-2.11.0
Detected OpenSearch Security Version: 2.11.0.0
 
Unable to find the admin password for the cluster. Please run 'export initialAdminPassword=<your_password>' or create a file /etc/opensearch/initialAdminPassword.txt with a single line that contains the password.

If you try to start the cluster it will then complain there is no certs ready as the demo certs are not copied due to above security demo scripts failures:

 
Caused by: org.opensearch.OpenSearchSecurityException: Error while initializing transport SSL layer from PEM: OpenSearchException[Unable to read /etc/opensearch/esnode.pem (/etc/opensearch/esnode.pem). Please make sure this files exists and is readable regarding to permissions. Property: plugins.security.ssl.transport.pemcert_filepath]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:482) ~[?:?]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:296) ~[?:?]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:202) ~[?:?]
        at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:232) ~[?:?]
        at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:278) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:782) ~[opensearch-2.11.0.jar:2.11.0]
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:731) ~[opensearch-2.11.0.jar:2.11.0]
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:533) ~[opensearch-2.11.0.jar:2.11.0]
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:195) ~[opensearch-2.11.0.jar:2.11.0]
        at org.opensearch.node.Node.<init>(Node.java:480) ~[opensearch-2.11.0.jar:2.11.0]
        at org.opensearch.node.Node.<init>(Node.java:407) ~[opensearch-2.11.0.jar:2.11.0]
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.11.0.jar:2.11.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.11.0.jar:2.11.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.11.0.jar:2.11.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.11.0.jar:2.11.0]
        ... 6 more
Caused by: org.opensearch.OpenSearchException: Unable to read /etc/opensearch/esnode.pem (/etc/opensearch/esnode.pem). Please make sure this files exists and is readable regarding to permissions. Property: plugins.security.ssl.transport.pemcert_filepath
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.checkPath(DefaultSecurityKeyStore.java:1133) ~[?:?]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.resolve(DefaultSecurityKeyStore.java:274) ~[?:?]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:452) ~[?:?]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:296) ~[?:?]
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:202) ~[?:?]
        at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:232) ~[?:?]
        at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:278) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) ~[?:?]
        at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
        at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) ~[?:?]
        at java.lang.reflect.Constructor.newInstance(Constructor.java:480) ~[?:?]
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:782) ~[opensearch-2.11.0.jar:2.11.0]
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:731) ~[opensearch-2.11.0.jar:2.11.0]
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:533) ~[opensearch-2.11.0.jar:2.11.0]
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:195) ~[opensearch-2.11.0.jar:2.11.0]
        at org.opensearch.node.Node.<init>(Node.java:480) ~[opensearch-2.11.0.jar:2.11.0]
        at org.opensearch.node.Node.<init>(Node.java:407) ~[opensearch-2.11.0.jar:2.11.0]
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-2.11.0.jar:2.11.0]
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-2.11.0.jar:2.11.0]
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.11.0.jar:2.11.0]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180) ~[opensearch-2.11.0.jar:2.11.0]
        ... 6 more

Tho this should fail the TAR as well, not sure why TAR is passing at the moment.

Attach systemd logs:

[ec2-user@ip-172-31-25-25 bin]$ sudo systemctl status -l opensearch.service
● opensearch.service - OpenSearch
   Loaded: loaded (/usr/lib/systemd/system/opensearch.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2023-10-06 23:14:10 UTC; 2 days ago
     Docs: https://opensearch.org/
  Process: 22264 ExecStart=/usr/share/opensearch/bin/systemd-entrypoint -p ${PID_DIR}/opensearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 22264 (code=exited, status=1/FAILURE)
 
Oct 06 23:14:10 ip-172-31-25-25.ec2.internal systemd-entrypoint[22264]: at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
Oct 06 23:14:10 ip-172-31-25-25.ec2.internal systemd-entrypoint[22264]: at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
Oct 06 23:14:10 ip-172-31-25-25.ec2.internal systemd-entrypoint[22264]: at org.opensearch.cli.Command.main(Command.java:101)
Oct 06 23:14:10 ip-172-31-25-25.ec2.internal systemd-entrypoint[22264]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
Oct 06 23:14:10 ip-172-31-25-25.ec2.internal systemd-entrypoint[22264]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
Oct 06 23:14:10 ip-172-31-25-25.ec2.internal systemd-entrypoint[22264]: For complete error details, refer to the log at /var/log/opensearch/opensearch.log
Oct 06 23:14:10 ip-172-31-25-25.ec2.internal systemd[1]: opensearch.service: main process exited, code=exited, status=1/FAILURE
Oct 06 23:14:10 ip-172-31-25-25.ec2.internal systemd[1]: Failed to start OpenSearch.
Oct 06 23:14:10 ip-172-31-25-25.ec2.internal systemd[1]: Unit opensearch.service entered failed state.
Oct 06 23:14:10 ip-172-31-25-25.ec2.internal systemd[1]: opensearch.service failed.

Thanks.

@DarshitChanpura
Copy link
Member

Hey @peterzhuamazon, we wouldn't expect it to fail this way because we already added the fix: opensearch-project/security#3490
Please open a new issue so we can track this. Also please add steps to reproduce this.
cc: @peternied @RyanL1997

@peterzhuamazon
Copy link
Member

Hey @peterzhuamazon, we wouldn't expect it to fail this way because we already added the fix: opensearch-project/security#3490 Please open a new issue so we can track this. Also please add steps to reproduce this. cc: @peternied @RyanL1997

Seems like the fix was merged after last RC generation.
Will generate again cc @rishabh6788 on this.

Thanks.

@prudhvigodithi
Copy link
Member

prudhvigodithi commented Nov 7, 2023

Hey @DarshitChanpura @RyanL1997 I can see there are couple of issues opened related to initial admin password change, can you please create one META issue and link them so that It would be easy to track and we can have all the conversation on that global META issue.
opensearch-project/security#3632
opensearch-project/security#3560
#4094
opensearch-project/security#3623
opensearch-project/security#3634
opensearch-project/security#3633
opensearch-project/security#3508

Also can you please provide the following details?

  • Details of the implementation
  • The impact on the distributions ?
  • Impact on OSD?
  • Impact on other open source platform for managing OpenSearch like docker, helm, ansible, kubernetes, terraform etc ?

@bbarani @peterzhuamazon @rishabh6788

@prudhvigodithi
Copy link
Member

prudhvigodithi commented Dec 5, 2023

Hey @DarshitChanpura @derek-ho @RyanL1997 following up on this issue to see if its still targeted for 2.12.0 release?
Adding @davidlago @bbarani @dblock

@DarshitChanpura
Copy link
Member

Closing this as all instances of admin credentials and requested changes are tracked as part this meta: opensearch-project/security#3624

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New Enhancement
Projects
Development

No branches or pull requests

6 participants