[BUG] Opensearch 2.17.0 Fails to start on Ubuntu 22.04.5 LTS (Jammy Jellyfish)

[servebeer]

Describe the bug

I have a "fresh" install of Ubuntu 22.04.5 LTS (Jammy Jellyfish), openjdk version "11.0.24", and installed Opensearch 2.17.0 (amd64) via APT. It installed with no errors but when I try to start it via "sudo systemctl start opensearch" it results in this error:

× opensearch.service - OpenSearch
Loaded: loaded (/lib/systemd/system/opensearch.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2024-10-21 17:30:43 UTC; 9min ago
Docs: https://opensearch.org/
Process: 628 ExecStart=/usr/share/opensearch/bin/systemd-entrypoint -p ${PID_DIR}/opensearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 628 (code=exited, status=1/FAILURE)
CPU: 19.110s

Oct 21 17:30:43 graylog systemd-entrypoint[889]: Error: A fatal exception has occurred. Program will exit.
Oct 21 17:30:43 graylog systemd-entrypoint[889]: at org.opensearch.tools.launchers.JvmErgonomics.flagsFinal(JvmErgonomics.java:125)
Oct 21 17:30:43 graylog systemd-entrypoint[889]: at org.opensearch.tools.launchers.JvmErgonomics.finalJvmOptions(JvmErgonomics.java:87)
Oct 21 17:30:43 graylog systemd-entrypoint[889]: at org.opensearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:70)
Oct 21 17:30:43 graylog systemd-entrypoint[889]: at org.opensearch.tools.launchers.JvmOptionsParser.jvmOptions(JvmOptionsParser.java:150)
Oct 21 17:30:43 graylog systemd-entrypoint[889]: at org.opensearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:108)
Oct 21 17:30:43 graylog systemd[1]: opensearch.service: Main process exited, code=exited, status=1/FAILURE
Oct 21 17:30:43 graylog systemd[1]: opensearch.service: Failed with result 'exit-code'.
Oct 21 17:30:43 graylog systemd[1]: Failed to start OpenSearch.
Oct 21 17:30:43 graylog systemd[1]: opensearch.service: Consumed 19.110s CPU time.

Related component

Build

To Reproduce

1. sudo systemctl status opensearch

2. Output matched error listed above

Expected behavior

Opensearch should start and run w/o any errors.

Additional Details

Plugins
none

Screenshots

Host/Environment (please complete the following information):

• OS: Ubuntu Jammy Jellyfish
• Version 22.04.5 LTS

Additional context
none

[servebeer]

I have also verified that /etc/opensearch/jvm.options has the following settings:
-Xms1g
-Xmx1g

[servebeer]

Here are the /etc/opensearch/opensearch.yml contents:

# ======================== OpenSearch Configuration =========================
#
# NOTE: OpenSearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.opensearch.org
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: graylog
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/opensearch
#
# Path to log files:
#
path.logs: /var/log/opensearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
# bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# OpenSearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
discovery.type: single-node
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of cluster-manager-eligible nodes:
#
#cluster.initial_cluster_manager_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
#
# ---------------------------------- Remote Store -----------------------------------
# Controls whether cluster imposes index creation only with remote store enabled
# cluster.remote_store.enabled: true
#
# Repository to use for segment upload while enforcing remote store for an index
# node.attr.remote_store.segment.repository: my-repo-1
#
# Repository to use for translog upload while enforcing remote store for an index
# node.attr.remote_store.translog.repository: my-repo-1
#
# ---------------------------------- Experimental Features -----------------------------------
# Gates the visibility of the experimental segment replication features until they are production ready.
#
#opensearch.experimental.feature.segment_replication_experimental.enabled: false
#
# Gates the functionality of a new parameter to the snapshot restore API
# that allows for creation of a new index type that searches a snapshot
# directly in a remote repository without restoring all index data to disk
# ahead of time.
#
#opensearch.experimental.feature.searchable_snapshot.enabled: false
#
#
# Gates the functionality of enabling extensions to work with OpenSearch.
# This feature enables applications to extend features of OpenSearch outside of
# the core.
#
#opensearch.experimental.feature.extensions.enabled: false
#
#
# Gates the optimization of datetime formatters caching along with change in default datetime formatter
# Once there is no observed impact on performance, this feature flag can be removed.
#
#opensearch.experimental.optimization.datetime_formatter_caching.enabled: false
#
# Gates the functionality of enabling Opensearch to use pluggable caches with respective store names via setting.
#
#opensearch.experimental.feature.pluggable.caching.enabled: false
#
# Gates the functionality of star tree index, which improves the performance of search aggregations.
#
#opensearch.experimental.feature.composite_index.star_tree.enabled: false

[servebeer]

I mis-spoke when I said that it installed with no issues. I just rebooted and noticed this:

Setting up opensearch (2.17.1) ...
Running OpenSearch Post-Installation Script
ERROR: Something went wrong during demo configuration installation. Please see the logs in /var/log/opensearch/install_demo_configuration.log
dpkg: error processing package opensearch (--configure):
installed opensearch package post-installation script subprocess returned error exit status 1
Processing triggers for libc-bin (2.35-0ubuntu3.8) ...
Errors were encountered while processing:
opensearch
E: Sub-process /usr/bin/dpkg returned an error code (1)

[servebeer]

I thought that the issue might be because I forgot to set an admin password at: /etc/environment I added the following line to it and rebooted:

OPENSEARCH_INITIAL_ADMIN_PASSWORD=StrongPassword

but I am still getting the error.

[servebeer]

So I thought that maybe I had to define the environment variable while installing opensearch, so I removed opensearch, then purged opensearch, and tried to reinstall with this command:

sudo env OPENSEARCH_INITIAL_ADMIN_PASSWORD=StrongPassword apt install opensearch

same errors... isn't there an easier way to enforce a password be set while installing?

[dbwiddis]

1. Try sudo -E env OPENSEARCH_INITIAL_ADMIN_PASSWORD=StrongPassword apt install opensearch
2. Does the strong password meet the complexity requirements?
   • Minimum 8 characters
   • Must contain at least one uppercase letter [A--Z]
   • One lowercase letter [a--z]
   • One digit [0--9]
   • One special character

[servebeer]

Same results:

Running OpenSearch Post-Installation Script
ERROR: Something went wrong during demo configuration installation. Please see the logs in /var/log/opensearch/install_demo_configuration.log
dpkg: error processing package opensearch (--configure):
installed opensearch package post-installation script subprocess returned error exit status 1
Processing triggers for libc-bin (2.35-0ubuntu3.8) ...
Errors were encountered while processing:
opensearch
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)

[dbwiddis]

I'm assuming the /var/log/opensearch/install_demo_configuration.log is what's telling you there's a password issue here?

@cwperks any ideas?

[cwperks]

If the post-installation script fails, you should be able to manually run it as well. The demo security configuration installer is located at <OPENSEARCH_HOME>/plugins/opensearch-security/tools. You can invoke it manually with ./install_demo_configuration.sh and answer all of the prompts, but make sure to set OPENSEARCH_INITIAL_ADMIN_PASSWORD before invoking the script. You can get real-time output when running the script and adjust accordingly if the pw does not meet complexity requirements.

[peterzhuamazon]

I dont have an environment for 22.04.5 for now.
Tho I also want to mention you dont need to install JDK11.
As the deb pkg is already including a copy of jdk.

[peterzhuamazon]

No issues on the latest 2.17.1:

$ cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.5 LTS"
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

$ sudo env OPENSEARCH_INITIAL_ADMIN_PASSWORD=myStrongPassword123! apt-get install opensearch
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  opensearch
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 906 MB of archives.
After this operation, 1179 MB of additional disk space will be used.
Get:1 https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable/main amd64 opensearch amd64 2.17.1 [906 MB]
Fetched 906 MB in 24s (37.3 MB/s)
Selecting previously unselected package opensearch.
(Reading database ... 96382 files and directories currently installed.)
Preparing to unpack .../opensearch_2.17.1_amd64.deb ...
Running OpenSearch Pre-Installation Script
Unpacking opensearch (2.17.1) ...
Setting up opensearch (2.17.1) ...
Running OpenSearch Post-Installation Script
### NOT starting on installation, please execute the following statements to configure opensearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable opensearch.service
### You can start opensearch service by executing
 sudo systemctl start opensearch.service
### Create opensearch demo certificates in /etc/opensearch/
 See demo certs creation log in /var/log/opensearch/install_demo_configuration.log
### Breaking change in packaging since 2.13.0
 In 2.13.0 and later releases of OpenSearch, we have changed the permissions associated with access to installed files
 If you are configuring tools that require read access to the OpenSearch configuration files, we recommend you add the user that runs these tools to the 'opensearch' group
 For more information, see https://github.com/opensearch-project/opensearch-build/pull/4043

[peterzhuamazon]

$ sudo systemctl start opensearch
$ curl https://localhost:9200 -ku admin:myStrongPassword123!
{
  "name" : "ip-172-31-43-194",
  "cluster_name" : "opensearch",
  "cluster_uuid" : "DT1wTm-kQi-sgc00WU2Cjg",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.17.1",
    "build_type" : "deb",
    "build_hash" : "1893d20797e30110e5877170e44d42275ce5951e",
    "build_date" : "2024-09-26T21:59:47.005429436Z",
    "build_snapshot" : false,
    "lucene_version" : "9.11.1",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

[peterzhuamazon]

Test 2.17.0, cleanup /etc/opensearch beforehand:

$ sudo env OPENSEARCH_INITIAL_ADMIN_PASSWORD=myStrongPassword123! apt-get install opensearch=2.17.0
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  opensearch
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 906 MB of archives.
After this operation, 1179 MB of additional disk space will be used.
Get:1 https://artifacts.opensearch.org/releases/bundle/opensearch/2.x/apt stable/main amd64 opensearch amd64 2.17.0 [906 MB]
Fetched 906 MB in 17s (52.0 MB/s)
Selecting previously unselected package opensearch.
(Reading database ... 96382 files and directories currently installed.)
Preparing to unpack .../opensearch_2.17.0_amd64.deb ...
Running OpenSearch Pre-Installation Script
Unpacking opensearch (2.17.0) ...
Setting up opensearch (2.17.0) ...
Running OpenSearch Post-Installation Script
### NOT starting on installation, please execute the following statements to configure opensearch service to start automatically using systemd
 sudo systemctl daemon-reload
 sudo systemctl enable opensearch.service
### You can start opensearch service by executing
 sudo systemctl start opensearch.service
### Create opensearch demo certificates in /etc/opensearch/
 See demo certs creation log in /var/log/opensearch/install_demo_configuration.log
### Breaking change in packaging since 2.13.0
 In 2.13.0 and later releases of OpenSearch, we have changed the permissions associated with access to installed files
 If you are configuring tools that require read access to the OpenSearch configuration files, we recommend you add the user that runs these tools to the 'opensearch' group
 For more information, see https://github.com/opensearch-project/opensearch-build/pull/4043

$ sudo systemctl start opensearch
$ curl https://localhost:9200 -ku admin:myStrongPassword123!
{
  "name" : "ip-172-31-43-194",
  "cluster_name" : "opensearch",
  "cluster_uuid" : "iYQGjo7PQ9mMSmYa3uujmg",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.17.0",
    "build_type" : "deb",
    "build_hash" : "8586481dc99b1740ca3c7c966aee15ad0fc7b412",
    "build_date" : "2024-09-13T01:04:13.864000718Z",
    "build_snapshot" : false,
    "lucene_version" : "9.11.1",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

[peterzhuamazon]

No issues with the default settings, able to install both 2.17.0 and 2.17.1 on Ubuntu 22.04.5 and start cluster without issues.

Thanks.

[servebeer]

@dbwiddis you are correct about the error coming from /var/log/opensearch/install_demo_configuration.log. I cleaned up a bunch of artifacts from previous install attempts and ran the install again with: sudo -E env OPENSEARCH_INITIAL_ADMIN_PASSWORD=StrongPassword apt install opensearch. This time, I got a different error: Running OpenSearch Post-Installation Script
ERROR: Something went wrong during demo configuration installation. Please see the logs in /var/log/opensearch/install_demo_configuration.log
dpkg: error processing package opensearch (--configure):
installed opensearch package post-installation script subprocess returned error exit status 1
Processing triggers for libc-bin (2.35-0ubuntu3.8) ...
Errors were encountered while processing:
opensearch

I think I am going to start from scratch and rebuild the virtual host.

[servebeer]

I appreciate everyone's input! I tried again after a fresh install of Ububtu Server, and this time it installed w/o any issues... go figure.