[BUG] Your request contained invalid JSON: 'utf-8' codec can't decode byte 0xeb (OpenAI connector)

[acha5066]

What is the bug?
When attempting to return an embedding for a string containing the character ë I get the error "Your request contained invalid JSON: 'utf-8' codec can't decode byte 0xeb in position 443". When I remove the character it works. I am retrieving the embedding from the _predict endpoint:

POST http://localhost:9200/_plugins/_ml/models/{model_id/_predict
{
    "parameters": {
        "input": ["This is a string containing Moët Hennessy"]
    }
}

Removing the special character and replacing it with e works. If I request the embedding directly from open AI (with special character) it works fine.

POST https://api.openai.com/v1/embeddings

{
    "input": "This is a string containing Moët Hennessy",
    "model": "text-embedding-ada-002",
    "encoding_format": "float"
}

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Setup Open AI connector described here:
2. Retrieve embedding for string described above.

What is the expected behavior?
Embedding should be returned for strings containing special characters.

What is your host/environment?

• Ubunutu
• Opensearch 2.11 running in a local docker setup.

Do you have any screenshots?
N/A

Do you have any additional context?
N/A

[ylwu-amzn]

Thanks for reporting this issue. Will fix this in PR #1691