WebSocket Connection Error with Gunicorn and Multiple Workers

[huoxiao-C]

After mounting Chainlit to the FastAPI application, if the Gunicorn command is used to start the application and multiple processes are specified (using the -w option), an error will occur. The startup command is: gunicorn -w 2 -k uvicorn.workers.UvicornWorker main:app --bind 0.0.0.0:3010

bug description

If the method based on FastAPI provided on the Chainlit official website (https://docs.chainlit.io/deploy/api) is used to mount Chainlit, and then the application is started using the Gunicorn command with multiple processes specified, such as: gunicorn -w 2 -k uvicorn.workers.UvicornWorker main:app --bind 0.0.0.0:3010, an error will be encountered on the web UI stating "Unable to connect to server".

Additionally, the console will display the following errors:

[2024-09-24 11:03:23 +0800] [10116] [INFO] connection closed
[2024-09-24 11:03:24 +0800] [10116] [INFO] ('127.0.0.1', 63124) - "WebSocket /chat/ws/socket.io/?EI0=4&transport=websocket&sid=TVH1xbI1HXjNDocwAAAK" 403
[2024-09-24 11:03:24 +0800] [10116] [INFO] connection rejected (403 Forbidden)
[2024-09-24 11:03:24 +0800] [10116] [INFO] connection closed
[2024-09-24 11:03:25 +0800] [10116] [INFO] ('127.0.0.1', 63129) - "WebSocket /chat/ws/socket.io/?EI0=4&transport=websocket&sid=i_R_YcNkA9SIqMBuAAAc" 403
[2024-09-24 11:03:25 +0800] [10116] [INFO] connection rejected (403 Forbidden)
[2024-09-24 11:03:25 +0800] [10116] [INFO] connection closed

Chainlit mounting code:

app = FastAPI()
@app.get("/")
def read_root():
    return {"message": "Hello, World!"}

mount_chainlit(app=app, target="src/chainlit_app.py", path="/chat").

software information

• chainlit：1.1.402
• python：3.8.5
• Browser：google chrome
• OS：windows or linux

[dosubot]

Hello, @huoxiao-C! I'm here to help you with any bugs, questions, or contributions you have regarding Chainlit. Let's tackle this issue together!

I found a similar closed issue that might be relevant to your problem: Error Message "Could not reach the server" on mounting to FastAPI. The issue was resolved by pull request #1337, which addressed WebSocket connection issues when submounting Chainlit.

The specific changes introduced in pull request #1337 include modifications to the mount_chainlit function in backend/chainlit/utils.py. These changes ensure that the environment variables CHAINLIT_ROOT_PATH and CHAINLIT_SUBMOUNT are set correctly, and the Chainlit app is properly mounted to the specified path. Here is the relevant code snippet:

def mount_chainlit(app: FastAPI, target: str, path="/chainlit"):
    os.environ["CHAINLIT_ROOT_PATH"] = path
    os.environ["CHAINLIT_SUBMOUNT"] = "true"
    from chainlit.config import config, load_module
    from chainlit.server import app as chainlit_app

    config.run.root_path = path
    config.run.debug = os.environ.get("CHAINLIT_DEBUG", False)

    check_file(target)
    # Load the module provided by the user
    config.run.module_name = target
    load_module(config.run.module_name)

    ensure_jwt_secret()

    app.mount(path, chainlit_app)

These changes ensure that the WebSocket connections work correctly when Chainlit is submounted ^[1][2].

Additionally, there are specific configurations required in Gunicorn to support WebSocket connections for Chainlit. The relevant settings are managed through Uvicorn, which is used to run the Chainlit server. Here are the key configurations:

1. WebSocket Protocol: The ws_protocol environment variable can be set to specify the WebSocket protocol. By default, it is set to "auto".

   ws_protocol = os.environ.get("UVICORN_WS_PROTOCOL", "auto")

2. Per-Message Deflate: The ws_per_message_deflate environment variable can be set to enable or disable per-message deflate. By default, it is enabled.

   ws_per_message_deflate_env = os.environ.get("UVICORN_WS_PER_MESSAGE_DEFLATE", "true")
   ws_per_message_deflate = ws_per_message_deflate_env.lower() in ["true", "1", "yes"]

These settings are then passed to the Uvicorn configuration when starting the server:

config = uvicorn.Config(
    app,
    host=host,
    port=port,
    ws=ws_protocol,
    log_level=log_level,
    ws_per_message_deflate=ws_per_message_deflate,
    ssl_keyfile=ssl_keyfile,
    ssl_certfile=ssl_certfile,
)

To summarize, you need to ensure that the UVICORN_WS_PROTOCOL and UVICORN_WS_PER_MESSAGE_DEFLATE environment variables are set appropriately to support WebSocket connections in Chainlit ^[3].

If you continue to experience issues, please verify that you are using the latest version of Chainlit and that the environment variables are correctly set.

To continue talking to Dosu, mention @dosu.

[dokterbob]

@dosu Could you suggest a more concise title for the issue?

@huoxiao-C Do let us know whether the bot helped you out!

[dosubot]

A more concise title for the issue could be:

"WebSocket Connection Error with Gunicorn and Multiple Workers" ^[1].

To continue talking to Dosu, mention @dosu.

[huoxiao-C]

hello dokterbob，the bot method is not ok，we still cannot specify multiworker with gunicorn

[dokterbob]

@huoxiao-C Sorry it took a while.

Could you perhaps increase logging to provide a full traceback for the 403? Also, make sure you're on the latest version (we improved some things around error handling).

We need to know the exact code generating this error.

Should work roughly like this: https://search.brave.com/search?q=fastapi+logging+of+tracebacks&source=web&summary=1&conversation=7c7357e15d8a0fc3f88da7&summary_og=bb464324cb55a1e106371f

Could you also double check (if you haven't already) that -w 1 doesn't generate the error?

[huoxiao-C]

@dokterbob hello dokterbob, thanks for your reply
1）There is no problem with setting the number of Gunicorn workers to 1.
2）I tried the method you mentioned, the message is as flows. The web UI still show that connection to the server failed，

and the backend server shows

。

This is my testing code

main.py

import traceback
from chainlit.utils import mount_chainlit
from fastapi import FastAPI, HTTPException
from fastapi.exceptions import RequestValidationError
from fastapi.responses import PlainTextResponse
from starlette.exceptions import HTTPException as StarletteHTTPException

app = FastAPI()


@app.exception_handler(StarletteHTTPException)
async def http_exception_handler(request, exc):
    print(f"Exception: {exc}, Traceback: {exc.__traceback__}")
    print(f"new Traceback：", ''.join(traceback.format_tb(exc.__traceback__)))
    return PlainTextResponse(str(exc), status_code=exc.status_code)

mount_chainlit(app=app, target="cl_test.py", path="/chat")

cl_test.py

import chainlit as cl
@cl.on_chat_start
async def on_chat_start():
    print("on_chat_start调用")
    guide_text = """Hi，我是DataAgent\n"""
    await cl.Message(guide_text).send()

app run command
gunicorn -w 2 -k uvicorn.workers.UvicornWorker main:app --bind 0.0.0.0:3010

software information
os：linux
python：3.9.20
chainlit：1.3.2
gunicorn：23.0.0