Fix failures to compute embeddings too much context & Shorten RAG Results

[jlewi]

Fix failures to compute embeddings because of computing too much context.

• Prior to this document we were computing the embeddings using the entire notebook.
  This would lead to context exceeded errors on longer documents.

• This had two negative impacts

  1. We stop learning from long documents because we no longer compute embeddings for the document
  2. When making suggestions we don't embed up retrieving any documents from RAG because we can't compute the embeddings
     for the current document

Don't include the full Document in the RAG example

• In Example.Query we were including the full document which would then be injected into the context when generating new suggests

• This can end up using a lot of tokens and potentially confusing the agent when generating new suggestions

• Use a simple algorithm to shorten the example. The aglorithm is as follows

  • Include the current selected cell
  • Keep including previous cells as long as they are markup cells

Results

• On our evaluation results the number of match cells is approximately the same; 6 correct whereas we got 7 before prior to this change
• Of these only 4 examples are the same ones that the two experiments both got correct
• We should probably add level 1 assertions to test whether results are retrieved to identify bugs like this in the future.

Other

• This PR also refactors the code to share code for computing embeddings between the learner and the Agent
  to minimize risk of training and serving skew.

• Treat learner failures as permanent failures rather than retrying. If we see concrete examples of retryable errors than we can add retries

  • Right now we were retrying on permanent errors (i.e. the context length being exceeded).

• Fix Error Learning From Examples: Can't compute embedding because of too much context #260

[netlify]

✅ Deploy Preview for foyle canceled.

Name	Link
🔨 Latest commit	b426981
🔍 Latest deploy log	https://app.netlify.com/sites/foyle/deploys/67032f516f7cc70008d62daf