AIQuality Really Drops Off When There is a lot of context

[jlewi]

Here's a long notebook
https://gist.github.com/jlewi/31857545ec62b36d2949ccd904918d53

The final markdown cell contained the following prompt

Fetch the trace 6518ae12b7aa41271978d9da16403df2 and render the request as HTML
Use the GetLLMLogs endpoint to fetch it and write it to a file
Then use Foyle llms render to render it as HTML

The LLM gave a really terrible response. It ended up giving a curl command with GetTrace which is a non-existent RPC.
The generated block id was 01J9545C1FHMBD9YNHFYTZGGSY.

I think the problem is all of the context (both in the notebook itself) and also likely retrieved examples is confusing the AI.

[jlewi]

I think this could be partially due to the fact that our embeddings computation

foyle/app/pkg/oai/embeddings.go

Line 22 in e2feb99

func (v *Vectorizer) Embed(ctx context.Context, text string) (*mat.VecDense, error) {

Tries to compute the embeddings of the full notebook. If this exceeds the context then know embeddings are retrieved and we don't get any RAG results. So for really long learning we don't benefit from RAG.