FS-60: Add promptfoo test config for suggestions prompt #14
Conversation
backend/src/prompts/README.md
Outdated
| Then run `promptfoo eval` to run promptfooconfig.yaml | ||
| ro run a specific file for example `generate_message_suggestions_config.yaml` run `promptfoo eval -c generate_message_suggestions_config.yaml` | ||
|
|
||
| Afterwards, you can view the results by running `promptfoo view` |
There was a problem hiding this comment.
Should these live under backend/tests/prompts instead? ... to be honest, I don't know what the tests in there are doing, or how they are supposed to be run...
It's becoming a bit of a mix...
There was a problem hiding this comment.
The tests we have under "prompts" already aren't doing a great deal in my opinion. Looks to me like it's just duplicating the code.
e.g. if someone was to make a change to the prompt file, the test would fail and then they would update the prompt exactly the same in the prompts file. I suppose it might alert you if you changed a prompt that you didn't realise was being nested inside another prompt somewhere else?
I would be tempted to scrap all the tests under /backend/tests/prompts and replace it with these...
There was a problem hiding this comment.
That being said, the tests in that folder are python based, and this is an npm based test. So maybe mixing them isn't a good idea. Possibly we could have it under /backend/promptfoo, I'm not sure...
There was a problem hiding this comment.
Because of how python does import you can't do that - at least I couldn't figure out how to
the problem line is from prompting import PromptEngine from get_generate_message_suggestions_prompt.py
can show you if you want
There was a problem hiding this comment.
I've made some updates to fix some of these issues
Co-authored-by: Charlie Leopard <[email protected]>
…v file. Simplify test structure so less utility .py classes are needed
…cottLogic/InferESG into FS-60/set-up-promptfoo-example
|
If someone can test my changes on their machine before we merge, that would be ideal |
Description
https://scottlogic.atlassian.net/browse/FS-60
I have added an example of promptfoo testing on the suggestions prompt with the hope that this can easily be modified to work for other prompts when we come to work on those.