FTL unable to create a watcher for configuration directory

[janaxhell]

Just deployed the latest build on my OMV6 server via docker-compose on Portainer and after downloading the models, LocalAI stops with this message:

## Send the example audio file to the transcriptions endpoint
curl http://localhost:8080/v1/audio/transcriptions \
     -H "Content-Type: multipart/form-data" \
     -F file="@$PWD/gb1.ogg" -F model="whisper-1"
})
4:59PM DBG Extracting backend assets files to /tmp/localai/backend_data
4:59PM DBG processing api keys runtime update
4:59PM DBG processing external_backends.json
4:59PM DBG external backends loaded from external_backends.json
4:59PM FTL unable to create a watcher for configuration directory error="too many open files" configdir=/build/configuration

So I cannot use it at all. Why does this happen? How can I fix it?

This is my compose:

version: "3.9"
services:
  api:
    image: localai/localai:latest-aio-cpu
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:7337/readyz"]
      interval: 1m
      timeout: 20m
      retries: 5
    ports:
      - 7337:8080
    environment:
      - DEBUG=true
    volumes:
      - /srv/dev-disk-by-uuid-8998116d-e145-4319-8c9f-41609747f657/Configs/localai/models:/build/models:cached
    
networks:
  default:
    external:
      name: omv2b

BTW what is the name of Stable diffusion 3 yaml file to add?

[mudler]

mmm while I can't reproduce this here, this sounds like a bug as shouldn't be a fatal error - care to open an issue so we can tackle it from there?

[mudler]

nevermind, good catch that was an oversight - fix in #2410 @janaxhell

[janaxhell]

Ehmm, that was fast. You mean I just have to update container?

[mudler]

the fix is going to be in the master images soon, will cut a patch release as follow up!

[janaxhell]

I suppose you still haven't fixed this. I've just deployed the latest build and while this time it doesn't exit, the container hangs as "Starting" in Portainer with this error:

wget --quiet --show-progress -O gb1.ogg https://upload.wikimedia.org/wikipedia/commons/1/1f/George_W_Bush_Columbia_FINAL.ogg
## Send the example audio file to the transcriptions endpoint
curl http://localhost:8080/v1/audio/transcriptions \
     -H "Content-Type: multipart/form-data" \
     -F file="@$PWD/gb1.ogg" -F model="whisper-1"
})
3:27PM DBG Extracting backend assets files to /tmp/localai/backend_data
3:27PM DBG processing api keys runtime update
3:27PM DBG processing external_backends.json
3:27PM DBG external backends loaded from external_backends.json
3:27PM ERR error establishing configuration directory watcher error="too many open files"
3:27PM INF core/startup process completed!
3:27PM DBG No configuration file found at /tmp/localai/upload/uploadedFiles.json
3:27PM DBG No configuration file found at /tmp/localai/config/assistants.json
3:27PM DBG No configuration file found at /tmp/localai/config/assistantsFile.json
3:27PM INF LocalAI API is listening! Please connect to the endpoint for API documentation. endpoint=http://0.0.0.0:8080

[mudler]

this looks like a log of an instance that started correctly - you should be able to hit the service already

[janaxhell]

The service page says:

{
  "error": {
    "code": 431,
    "message": "Request Header Fields Too Large",
    "type": ""
  }
}

[janaxhell]

Any idea? The compose is exactly the same, I just redeployed it forcing the pull of new build.

[janaxhell]

Please, can you help me? I've been waiting for a month to try this and all I found about "Request Header Fields Too Large" doesn't help me in any way. The container looks healthy, the log as well, I'm one step from using it and I cannot access the UI.

4:13PM INF LocalAI API is listening! Please connect to the endpoint for API documentation. endpoint=http://0.0.0.0:8080
4:14PM INF Success ip=172.32.0.1 latency="819.368µs" method=GET status=200 url=/readyz
4:15PM INF Success ip=172.32.0.1 latency=1.517497ms method=GET status=200 url=/readyz
4:16PM INF Success ip=172.32.0.1 latency="30.787µs" method=GET status=200 url=/readyz
4:17PM INF Success ip=172.32.0.1 latency="27.007µs" method=GET status=200 url=/readyz
4:18PM INF Success ip=172.32.0.1 latency="39.801µs" method=GET status=200 url=/readyz
4:19PM INF Success ip=172.32.0.1 latency="34.674µs" method=GET status=200 url=/readyz
4:20PM INF Success ip=172.32.0.1 latency="54.276µs" method=GET status=200 url=/readyz
4:21PM INF Success ip=172.32.0.1 latency="51.148µs" method=GET status=200 url=/readyz
4:22PM INF Success ip=172.32.0.1 latency="44.097µs" method=GET status=200 url=/readyz
4:23PM INF Success ip=172.32.0.1 latency="36.355µs" method=GET status=200 url=/readyz
4:24PM INF Success ip=172.32.0.1 latency="45.039µs" method=GET status=200 url=/readyz
4:25PM INF Success ip=172.32.0.1 latency="27.887µs" method=GET status=200 url=/readyz
4:26PM INF Success ip=172.32.0.1 latency="52.337µs" method=GET status=200 url=/readyz