Ongoing problems with ot-es container

[kenibrewer]

Hi @mbdebian

Just following up on #6. The recent changes you pushed failed to resolve the issues that we were having with the ot-es container on Ubuntu 24.04.01.

Attached are the log files for ot-es container on our deployment which is a clean EC2 instance built from the standard AWS Ubuntu 24.04 AMI with docker, docker compose and gsutil their dependencies as the only additional tools installed on the image.

ot-es.log

The part of the logs that stands out to me are the lines before and after the java.nio.file.NoSuchFileException errors.

Have you tested the deployment of the standalone-deployment-platform on a clean VM or machine image? If so could you provide some more detailed instructions about your testing environment so we could replicate it for our deployment?

Thanks again for making such an exciting tool accessible to the research community!

[javfg]

Hi Ken,

This looks like a different problem than #6, the error you get is related to opensearch security analytics (there seems to be something similar here). We'll investigate.

Regarding our testing, we are not using the standalone deployment in cloud environments or vms, just in our physical machines.

I'll keep you posted if we find anything!

[AnneCarpenter]

Hi! I wonder if you uncovered anything to solve this? We are stuck here :(

[javfg]

We're currently finishing some big changes in the platform and haven't been able to check on this yet.

But it is still in our radar, we will get to it.

Sorry for the delay!

[mbdebian]

Hi @kenibrewer, please let me have a look at the logs and try to reproduce your environment, I'll get back to you along this week

[javfg]

Hi @kenibrewer @AnneCarpenter, we're looking into it at the moment.

Meanwhile, if you are running things manually, you could try updating opensearch to the latest version in the profile you are using by changing:

export OTOPS_PROFILE_ELASTIC_SEARCH_DOCKER_IMAGE_TAG=2.10.0

to

export OTOPS_PROFILE_ELASTIC_SEARCH_DOCKER_IMAGE_TAG=2.18.0

then giving it another try. Please do this in a new machine or clean up all containers and reset the profile before:

sudo make clean
make set_profile profile=<profile_you_are_using>
make platform_up

If you do so, report back to us with the result.

[mbdebian]

Hi @kenibrewer , please, have a look at #6 where I've made changes in a branch that I think will probably address your problem when deploying open targets in EC2.

I've verified those changes for a 22.04.5 LTS, but they're very likely to apply to your EC2 based Ubuntu 24.04.01.

Would you mind giving them a try while I set up another environment to mimic yours, so I can find out whether there may be differences between those Ubuntu versions that need to be addressed for the stand alone deployment tool to work?

Thanks @kenibrewer !

[mbdebian]

Hi @kenibrewer , I've just verified that the branch local_issue_6_and_8 works with the following system:

Distributor ID: Ubuntu
Description:    Ubuntu 24.04.1 LTS
Release:        24.04
Codename:       noble

I have used profile 2406 to make sure the data backend tarballs for Clickhouse and Opensearch are accessible and work.

[kenibrewer]

Hi @mbdebian the fixes on the branch local_issue_6_and_8 worked in resolving the issues with the ot-es container.