[Security solution] Elastic Assistant, OpenAI stream

[stephmilovic]

Part 1 of streaming in Security's AI Assistant

Implements streaming for OpenAI in a non-langchain environment

To test:

1. Have an OpenAI connector
2. Update the feature flag to be true. Located in kbn-elastic-assistant/impl/assistant/api.tsx named isStream
3. Test chatting with the AI Assistant. Test Stop generating button. Test Regenerate button

To do in follow up:

1. Add support for langchain
2. Bedrock streaming
3. Remove isStream feature flag

[spong]

Is this interface intended to stay the same through part2? Will we support streaming turned on/off (even if just technically and not exposed via the UI)? If so, we might want to rework these types so one of the two are required. Does a message make sense without content now? I suppose the new empty 'loading' message would fit this interface?

[stephmilovic]

if a message has both content + reader, we know to show the "Regenerate response" button. @jamesspi thinks we should turn streaming on 100% of the time. Even with streaming, once a reader is complete the message will get its content property

[andrew-goldstein]

The description of this test appears to contain three assertions, and the implementation contains a total of 5 expects. Consider (generally) one assertion per test

[spong]

Looks like everything around the markdown parsing/processing got moved over correctly, however I'm not seeing the codeblock actions show up anymore:

I confirmed the custom codeblock is rendering as expected, so it is probably something downstream w/ augmentMessageCodeBlocks

kibana/x-pack/plugins/security_solution/public/assistant/helpers.tsx

Lines 80 to 97 in 480ef87

	/**
	* Augments the messages in a conversation with code block details, including
	* the start and end indices of the code block in the message, the type of the
	* code block, and the button to add the code block to the timeline.
	*
	* @param currentConversation
	*/
	export const augmentMessageCodeBlocks = (
	currentConversation: Conversation
	): CodeBlockDetails[][] => {
	const cbd = currentConversation.messages.map(({ content }) =>
	analyzeMarkdown(
	getMessageContentWithReplacements({
	messageContent: content ?? '',
	replacements: currentConversation.replacements,
	})
	)
	);

You had mentioned some changes around getMessageContentWithReplacements(), which this ends up calling -- perhaps it's coming from there?

[stephmilovic]

Ok I've got this sort of figured out. First off, I dropped the className that the getControlContainer was looking for, className={'message-${index}'}. I put the class in place, and it is working again for NON-streaming.

Putting a debugger in the getControlContainer while streaming is turned on, when our comments changes from reader to content there seems to be a flash where the .euiCodeBlock__controls disappears. As soon as the debugger is played, the controls appear again. ❓

Playing with this, I tried switching from useLayoutEffect to useEffect, but the same issue happened. I added a setTimeout, and experimented with the timeout length and even a timeout of 0 seems to resolve the issue. Kind of a render bandaid that appears to be holding... what do you think? Any better ideas?

[spong]

Discussed offline w/ Steph -- this all looks good and even improves the inconsistencies with the current experience, so all the better! 🙂

[spong]

This might not be the relevant file (perhaps stream_observable), but commenting here instead of the main PR for thread support.

Anyway, I've somehow got myself in a state where all my replies end up being garbled as below.

While streaming vs. complete:

[spong]

I don't have exact repro steps for getting it consistently stuck, but at the time I was testing closing/opening the assistant mid-stream and then later opening/closing the timeline instance of the assistant while it was streaming as well.

[stephmilovic]

shit ok, i've not seen this. I bet calling useStream's setComplete callback on modal close will prevent this.

[stephmilovic]

I believe this should resolve the issue: cae753a

[andrew-goldstein]

Thanks for this first phase of streaming @stephmilovic! 🙏
LGTM 🚀

[stephmilovic]

@elasticmachine merge upstream

[stephmilovic]

@elasticmachine merge upstream

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 78bf5e2

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
securitySolution	4662	4670	+8

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/elastic-assistant	75	76	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	13.1MB	13.1MB	+14.4KB

Unknown metric groups

API count

id	before	after	diff
@kbn/elastic-assistant	95	96	+1

History

• 💔 Build #173677 failed 7b0d156
• 💔 Build #173665 failed 9a2330b
• 💔 Build #173650 failed be53e17
• 💛 Build #173273 was flaky a920258
• 💔 Build #173245 failed 61c6ed4

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream