The Search Query encounters a failure within the elastic search cluster

[wilmerism]

Afternoon, i have this issue in the subject line,

Is Security onion 2.4.111

Sudo so-status shows all services is green and running.

From the SO- Web interface under "grid" Elastic search status = FAULT in red

Ive googled it to death, ive then spent the last few hours on chatgpt diagnosing this... so i asked chat gpt to summarise what the issue is (it keeps on about certificate issue... ) and what we tried in the form of a question suitable for here can i get some pointers please. >>

Title: Unable to Verify SSL Certificate for Elasticsearch on Security Onion 2.4.111

Body:

Hello,

I am encountering an issue with Elasticsearch on Security Onion version 2.4.111. Here is a summary of the problem and the troubleshooting steps I have taken so far:

Original Issue
I noticed that Elasticsearch was not responding correctly to queries. Using curl resulted in an "Empty reply from server."
Upon testing Elasticsearch via curl on https://localhost:9200, I encountered an SSL-related error:
vbnet
Copy code
curl: (60) SSL certificate problem: unable to get local issuer certificate
What We Have Tried
Exported the Certificate: Using openssl, I extracted the Elasticsearch certificate:

bash
Copy code
echo | openssl s_client -connect localhost:9200 -showcerts > elasticsearch.crt
The certificate was successfully saved, but it showed the error:
kotlin
Copy code
Verify return code: 21 (unable to verify the first certificate)
Added the Certificate to Trusted CA Store:

Tried to copy the elasticsearch.crt to /usr/local/share/ca-certificates/ and run update-ca-certificates. However, the update-ca-certificates command is not available on Security Onion.
Manual Testing with Custom CA Bundle:

Created a custom CA bundle containing elasticsearch.crt and used it with curl:
bash
Copy code
curl --cacert ./elasticsearch.crt -X GET "https://localhost:9200"
The error persists, indicating the certificate chain or trust setup is incomplete.
Verified Disk and Service Status:

The so-elasticsearch container is running without visible errors in sudo so-status.
Disk space and memory are sufficient.
Questions
How can we properly configure SSL and the Elasticsearch certificate in Security Onion to resolve this issue?
Is there an alternative to update-ca-certificates for managing trusted certificates on Security Onion?
Could this be related to the default keystore (elasticsearch.p12) used by Elasticsearch? If so, how should we regenerate or replace it?
Evidence and Logs
Error from curl:
vbnet
Copy code
curl: (60) SSL certificate problem: unable to get local issuer certificate
Extracted certificate via openssl:
kotlin
Copy code
Verify return code: 21 (unable to verify the first certificate)
Logs from Security Onion show that the so-elasticsearch container is running, and keystore operations are successful:
bash
Copy code
Importing keystore /usr/share/elasticsearch/config/sokeys/elasticsearch.p12
Import command completed: 1 entries successfully imported
Any guidance or documentation references would be greatly appreciated!

[TOoSmOotH]

Did you look at the logs in /opt/so/log/elasticsearch?

[wilmerism]

HI,

i did have disk space issues, this was fixed by running >

sudo find /nsm/pcap -type f -mtime +90 -exec rm -f {} ;
sudo find /nsm/suricata -type f -mtime +90 -exec rm -f {} ;
sudo find /nsm/zeek -type f -mtime +90 -exec rm -f {} ;
sudo find /nsm/elasticsearch/indices -type d -mtime +90 -exec rm -rf {} ;

heres a screen shot of the output from sudo grep "error" /opt/so/log/elasticsearch/securityonion.log | head -n 1 (theres loads of them)

when i seach im getting >

From the additional logs provided, the repeated errors and the presence of Suricata-related logs (e.g., .ds-logs-suricata) suggest that the issue involves Elasticsearch being unable to properly manage or query indices related to Suricata. The error mentions missing shards and partial searches, indicating that Elasticsearch is struggling with data integrity or resource issues.

Root Causes
Corrupted or Missing Shards:

Some shards (segments of Elasticsearch indices) are missing or corrupted, which prevents Elasticsearch from fulfilling queries.
Storage or Memory Pressure:

Insufficient disk space or memory could cause indices to become incomplete or inaccessible.
Old or Misconfigured Indices:

Indices created over time may not align with the current system configuration.
Elasticsearch Not Properly Installed:

Previous logs suggest Elasticsearch is not properly installed or configured, further complicating index management.

Thanks ,

[TOoSmOotH]

You will need to reinstall:

sudo find /nsm/elasticsearch/indices -type d -mtime +90 -exec rm -rf {} ;

Always use the Elastic API to remove indices. You nuked some important system indices with that command. This is why elastic is complaining. You may be able to copy your old indices to a different directory and then import them back as dangling idices but that is a lot of work.

[wilmerism]

doh..
Those 4 commands worked fin on my lab setup, that security onion still works (although that one didnt have the issue) ??

Ill plumb for a reinstall, thanks

[TOoSmOotH]

Most likely didn't have any files older than 90 days on your lab