-
Notifications
You must be signed in to change notification settings - Fork 11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
run defence detection and chat completion concurrently #375
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One suggestion, and one concern to point out, but the logic seems sound.
I am of course concerned that changes such as these are not covered by tests!
wonLevel: false, | ||
}; | ||
const message = req.body.message; | ||
const currentLevel = req.body.currentLevel; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Destructuring is neater:
const { message, currentLevel } = req.body;
...openAiReply.defenceInfo.triggeredDefences, | ||
]; | ||
// combine blocked | ||
chatResponse.defenceInfo.isBlocked = openAiReply.defenceInfo.isBlocked; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In general, I'm a little concerned about this mutable chatResponse object being passed around, and modified as a side-effect, as it feels like a difficult-to-detect bug waiting to emerge, particularly in functions with high complexity such as these. I am having difficulty seeing all the places it can be modified, and in what order.
I'll add a future issue to tackle this complexity, and introduce immutability.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great change, this has sped things up a massive amount 🏃♀️
* run defence detection and chat completion concurrently * split up chat request * refactor error handling
this ticket
in the chat endpoint (in router.ts), split logic for level 1 and 2 (where no defences are applied), and level 2 and 3 (where we detect defences and can block message if turned on)
for level 2 and 3, call detectTriggeredDefences and chatGptSendMessage in parallel
when both are resolved, check if message was blocked in detectTriggeredDefences
if it is blocked, restore the chat history object from before the chat was sent (essentially undoing the chat message as we now find out it was blocked)
set the openAi reply to stop the message being returned to the user
testing
repeated this conversation in dev and this refactored version spy-logic-chat-log-sandbox.pdf
recorded time for whole endpoint to return with each input prompt refactor_time_comparison.ods
on average saving almost 5 seconds
comments