Cortex should wait for ES to be available

[priamai]

Hi there,
I noticed a critical issue with the Cortex container that doesn't check when ES is up and running.
It gets into a state where it doesn't generate the cortex_5 indexes at the first time and then fails the database migration:

[info] play.api.Play - Application started (Prod) (no global state)
[error] o.e.d.DBConfiguration - ElasticSearch request failure: POST:/cortex_5/_search?scroll=60000ms
StringEntity({"seq_no_primary_term":"true","query":{"bool":{"must":[{"term":{"relations":{"value":"worker"}}},{"match_all":{}}]}},"from":0,"sort":[{"_doc":{"order":"desc"}}]},Some(application/json))
 => ElasticError(index_not_found_exception,no such index [cortex_5],Some(_na_),Some(cortex_5),None,List(ElasticError(index_not_found_exception,no such index [cortex_5],Some(_na_),Some(cortex_5),None,null,None,None,None,List())),None,None,None,List())
[error] o.e.d.DBConfiguration - ElasticSearch request failure: POST:/cortex_5/_search?scroll=60000ms
StringEntity({"seq_no_primary_term":"true","query":{"bool":{"must":[{"term":{"relations":{"value":"job"}}},{"term":{"status":{"value":"Waiting"}}}]}},"from":0,"sort":[{"_doc":{"order":"desc"}}]},Some(application/json))
 => ElasticError(index_not_found_exception,no such index [cortex_5],Some(_na_),Some(cortex_5),None,List(ElasticError(index_not_found_exception,no such index [cortex_5],Some(_na_),Some(cortex_5),None,null,None,None,None,List())),None,None,None,List())
[warn] o.e.d.SearchWithScroll - Search error
org.elastic4play.IndexNotFoundException$: null
        at org.elastic4play.IndexNotFoundException$.<clinit>(Errors.scala)
        at org.elastic4play.database.DBConfiguration.$anonfun$execute$2(DBConfiguration.scala:155)
        at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
        at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
        at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:56)
        at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:93)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
        at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:93)

The best solution is to add wait-for-it in the docker image:

https://docs.docker.com/compose/startup-order/

I am a bit unsure about how to modify the command but a good starting point:

FROM thehiveproject/cortex:latest
COPY wait-for-it.sh /opt/cortex/wait-for-it.sh
RUN chmod +x /opt/cortex/wait-for-it.sh

And then maybe something like this:

  cortex:
    build:
      context: ./fixcortex
    container_name: cortexfix
    restart: unless-stopped
    command: ["/opt/cortex/wait-for-it.sh", "elasticsearch_thp:9200", "--", "job-directory ${JOB_DIRECTORY}"]
    environment:
      - 'JOB_DIRECTORY=${JOB_DIRECTORY}'
    volumes:
      - './vol/cortex/application.conf:/etc/cortex/application.conf'
      - './vol/cortex/jobs:${JOB_DIRECTORY}'
      - '/var/run/docker.sock:/var/run/docker.sock'
    depends_on:
      - elasticsearch_thp
    ports:
      - 9001:9001

[priamai]

I believe I am experiencing the same bug that was mentioned here a few years back