test: tweak smoke test tool bodies to standardize response text #876
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
thedadams
approved these changes
Oct 14, 2024
njhale
requested review from
thedadams,
drpebcak,
StrongMonkey,
ryanhopperlowe,
g-linville,
tylerslaton and
iwilltry42
njhale
changed the title
thedadams
previously approved these changes
Oct 14, 2024
njhale
requested review from
thedadams
and removed request for
iwilltry42 and
ryanhopperlowe
thedadams
reviewed
Oct 14, 2024
|
|
||
| --- | ||
| name: bob | ||
| description: I'm Bob, a friendly guy. | ||
| args: question: The question to ask Bob. | ||
|
|
||
| When asked how I am doing, respond with exactly "Thanks for asking "${QUESTION}", I'm doing great fellow friendly AI tool!" | ||
| When asked how I am doing, respond with the following exactly: "Thanks for asking '${question}'! I'm doing great fellow friendly AI tool!" with ${question} replaced with the question text as given. |
There was a problem hiding this comment.
I thought we changed this such that question should be QUESTION?
There was a problem hiding this comment.
yup, we did.
Fixed and pushed.
There was a problem hiding this comment.
curious if this makes any functional difference since this isnt a code tool - ${QUESTION} is being made to look like an environment variable here, but there isnt anything actually setting or reading env vars... its just the llm being a smarty pants.
There was a problem hiding this comment.
Yeah, this is more for convention sake than anything.
The extra text explaining how to "interpolate" the variable is because 4o isn't actually that much of a smarty pants after all
Tweak the tool bodies for smoke test GPTScripts to reduce ambiguity in the response. This prevents models -- like gpt-4o -- from doing things like failing to interpolate strings consistently between runs. Signed-off-by: Nick Hale <[email protected]>
Signed-off-by: Nick Hale <[email protected]>
njhale
force-pushed
the
test/smoke-tweak-tc-bodies
branch
from
1c95f69
to
1b6f172
Compare
thedadams
approved these changes
Oct 14, 2024
drpebcak
approved these changes
Oct 14, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Smoke tests flake for
gpt-4ob/c of non-determinism in how it interpreted the test case instructions (e.g. failed to interpolate a string variable consistently). This change reduces ambiguity in the tool instructions so that it produces consistent results across smoke test runs.Note: I regenerated golden files across all models and ran the tests 10 times per model to vet this change.